Encrypted Training
FHEON can fine-tune a network with the data, the activations and the parameters encrypted. Every layer has a forward-pass primitive and a backward-pass primitive:
the forward pass computes the layer on the encrypted batch and caches what its gradient needs (inputs, the normalized values of a batch norm, the downsampled phases of a stride-2 convolution, …);
the backward pass receives the gradient of the layer output, returns the gradient of its input and applies the SGD update
p <- p - lr * dL/dpto every encrypted parameter, which it writes back to disk.
A training step (see Training and Fine-Tuning Examples) runs the forward primitives layer after layer, computes the softmax
and the cross-entropy gradient, then runs the backward primitives in reverse order. The parameters and caches are
stored as serialized ciphertexts in --training-data-dir (default ./../encrypted_training_data/), so a step can
continue from the parameters of the previous one.
Standard layout
Training uses the standard layout: one ciphertext per channel, holding the batch image after image, and one ciphertext per scalar weight (broadcast over the slots).
Controllers |
FHEONTrainForwardPassController / FHEONTrainBackwardPassController, on FHEONTrainSupportController, and their multithreaded, optimized versions FHEONTrainStandardForwardPassController / FHEONTrainStandardBackwardPassController. |
ResNet-20 model |
|
Validation |
|
Forward-pass primitives
Layer |
Primitive ( |
|---|---|
3x3 convolution |
|
1x1 shortcut |
|
Batch normalization |
|
ReLU |
|
Pooling |
|
Linear |
|
Softmax |
|
Backward-pass primitives
The backward primitives are named after the forward ones (backward_pass_convolution,
backward_pass_batch_norm, backward_pass_relu, backward_pass_cross_entropy_and_softmax, …).
A stride-2 convolution upsamples its gradient back to the input resolution with a backward dilation. Three variants
exist: backward_dilate, backward_dilate_optimized (fewer operations) and backward_dilate_logdepth (the
adjoint of the forward downsampling, logarithmic depth). The multithreaded backward controller fuses each layer per
input channel, so every weight is read once, used for the input gradient and updated.
Numerics and parameters
The following choices determine the accuracy of encrypted training.
Batch normalization. 1/sqrt(var + eps) is computed by Newton’s method. A constant start only converges near
var = 1 / start^2, while the batch variances of a trained ResNet-20 span 1e-13 ... 2. The per-channel
overloads of he_forward_batch_norm therefore start every channel from 1 / sqrt(running_var + eps) of the
pretrained model (<layer>_running_var.csv, written by python/training/export_tuneresnet20_weights.py).
Newton runs on the normalized variance (var + eps) y0^2 (close to 1), with y0 applied only as a plaintext
factor, so every bootstrapped value stays small.
Softmax. 1 + x + x^2/2 is not monotonic for x < -1 and picks wrong classes on real logits, and
exp(x) itself overflows the CKKS range. he_forward_softmax_normalized computes q = exp(z / 2^r) and then
q <- q^2 / sum(q^2) r times. The values stay within [0, 1], and nothing is decrypted.
Bootstrapping precision. A 2-iteration bootstrapping only improves on one iteration when it is given the precision
of the first iteration: FHEONHEController::set_bootstrap_precision(15), or setBootstrapPrecision of the
training controllers. With the parameters of the training models (ring 2^15, first modulus 50 bits, scale 46 bits),
this brings the bootstrapping error from about 8e-3 to 3e-7.
Pretrained parameters. tuneResNet20 imports the CSV exports of a PyTorch checkpoint on its first run: the
parameters missing from --training-data-dir are read directly from the CSV files (pretrainedFromCsv).
Running training programs
Command line. Build with
-DMODE=TRAININGand runtuneResNet20(see Training and Fine-Tuning Examples).FHEON Studio. The Workbench runs
tuneResNet20under the task Training: choose a batch of 16 CIFAR-10 images and the number of worker threads; the run passes--training-data-dir(one folder per program,encrypted_training_data/studio-tuneResNet20/). See FHEON Studio.
The parameters are ciphertexts under the keys of the run that wrote them: a run that generates new keys cannot read the
parameters of an earlier run, so it starts from new parameters, while a run that loads the keys (--load-context)
continues from them.
Validation
HETrainingModels/TrainingModules.cpp (target trainModules) runs single backward primitives on deterministic
inputs, to be compared with python/training/validate_modules.py.