Standard Forward Pass Controller (Multithreaded)
Multithreaded, optimized version of FHEONTrainForwardPassController (same files and keys), with per-channel batch norm and the repeated-squaring softmax.
-
class FHEONTrainStandardForwardPassController : public FHEONTrainForwardPassController
Multithreaded forward pass of the training pipeline, standard layout (one channel of the batch per ciphertext).
Same primitives, arguments, cache files and results as FHEONTrainForwardPassController (it derives from it and keeps its helpers), with:
ciphertext weights multiplied without relinearization and relinearized once per output (the original relinearizes every product: inputChannels * 9 key switchings per output channel of a convolution, now 1);
every weight read from disk once (the original reads every convolution weight twice and writes the linear and shortcut weights back after reading them), streamed per output instead of held for the whole layer;
the global average pooling sums the images of a channel with one rotate-and-add (log2(w^2) rotations) instead of one EvalSum per image;
the batch norm keeps EvalBootstrap on the calling thread and runs the Newton iterations of all channels in lock step, with the first iteration in one level (y0 is a known constant);
the number of workers capped with setWorkerThreads, and the bootstrapping iterations set with setBootstrapIterations (2, as the original, by default). The rotations are those of the original, so the existing training key files apply unchanged.
Public Functions
-
inline FHEONTrainStandardForwardPassController(CryptoContext<DCRTPoly> ctx)
-
inline void setWorkerThreads(int numThreads)
-
inline int getWorkerThreads() const
-
inline void setBootstrapIterations(int iterations)
-
inline void setBootstrapPrecision(int bits)
-
vector<Ctext> he_forward_convolution_optimized(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int stride, string layerName = "", bool pretrainedFromCsv = false)
3x3 convolution (padding 1, stride 1 or 2) with ciphertext weights, see FHEONTrainForwardPassController::he_forward_convolution_optimized.
The taps of every input channel are computed in parallel (same rotations as the original), then every output channel accumulates its inputChannels * 9 products without relinearization and relinearizes once. Every weight is read once, in the worker of its output channel.
-
vector<Ctext> he_forward_convolution_shortcut_optimized(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int stride, string layerName = "", bool pretrainedFromCsv = false)
1x1 shortcut convolution (stride 1 or 2), see FHEONTrainForwardPassController::he_forward_convolution_shortcut_optimized.
Weights streamed per output channel and multiplied without relinearization; the weights are no longer written back after being read (deserialize_and_load already stores a weight it initializes).
-
vector<Ctext> he_forward_global_avgpool(vector<Ctext> &encryptedInputs, int batchSize, int dataWidth, int inputChannels, string layerName = "")
Global average pooling, see FHEONTrainForwardPassController::he_forward_global_avgpool.
One rotate-and-add over the spatial positions (log2(w^2) rotations, the EvalSum keys) leaves the sum of image b in slot b * w^2 for every image at once; the per-image sums are then brought to slot 0 with the rotations by w^2 of the original and merged. The original computed one EvalSum per image (batchSize * log2(w^2) rotations).
-
vector<Ctext> he_forward_relu(vector<Ctext> &encryptedInputs, vector<int> scaleValues, int inputChannels, int vectorSize, int polyDegree = 59, string layerName = "")
ReLU of every channel (Chebyshev approximation of the input normalized by 2 * scale), see FHEONTrainForwardPassController::he_forward_relu. The unused mask plaintext of the original is not encoded.
-
vector<Ctext> he_forward_avgPool_optimized(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int kernelWidth, int stride = 2, string layerName = "")
2x2 average pooling, see FHEONTrainForwardPassController::he_forward_avgPool_optimized (window sums with hoisted rotations, then the downsampling of every channel in parallel).
-
vector<Ctext> he_forward_linear_multiple_outputs(vector<Ctext> &encryptedInput, int vectorSize, int inputSize, int outputSize, string layerName = "", bool pretrainedFromCsv = false)
Fully connected layer with one ciphertext per neuron, see FHEONTrainForwardPassController::he_forward_linear_multiple_outputs.
Weights streamed per output neuron (the original holds the whole outputSize x inputSize matrix and writes every weight back after reading it), products accumulated without relinearization.
-
vector<Ctext> he_forward_softmax(vector<Ctext> &encryptedInputs, const vector<int> &scalingValues, int inputSize, int appxPolyDegree = 59)
Softmax exp(x_i) / sum_j exp(x_j) with exp(x) ~ 1 + x + x^2 / 2 and the reciprocal from Newton-Raphson on 1/sqrt (then squared), see FHEONTrainForwardPassController::he_forward_softmax.
-
vector<Ctext> he_forward_batch_norm(vector<Ctext> &encryptedInputs, int inputSize, double appxInitValue, int appxDegree = 3, string layerName = "", bool isBootstrap = true, bool pretrainedFromCsv = false)
Batch normalization of every channel over its inputSize slots, see FHEONTrainForwardPassController::he_forward_batch_norm.
Statistics of all channels in parallel; the bootstrappings on the calling thread (EvalBootstrap parallelizes itself with OpenMP); Newton iterations of all channels in lock step; scale, shift and caches in parallel. Same cache files.
-
vector<Ctext> he_forward_batch_norm(vector<Ctext> &encryptedInputs, int inputSize, const vector<double> &channelInverseStdInit, int appxDegree, string layerName, bool isBootstrap, bool pretrainedFromCsv, double epsilon = 1e-5)
Batch norm with a per-channel start of Newton’s 1/sqrt(var + epsilon): channelInverseStdInit[c] (e.g. 1 / sqrt(running_var[c] + eps) of the pretrained model). A constant start (appxInitValue) only converges near var = 1 / start^2, and the batch variances of a trained ResNet20 span 1e-13 … 2: 3 iterations leave 1/std up to 1000x too small, which shrinks the activations layer after layer. Newton runs on the normalized variance (var + eps) start^2 ~ 1 from 1 and the start enters only as scalar products, so the bootstrapped values stay small (1/std reaches 1/sqrt(eps) ~ 316 on dead channels). Same caches and parameter files as the overload above.
-
vector<Ctext> he_forward_softmax_normalized(vector<Ctext> &logits, int batchSize, double logitBound = 32.0, int finalNewtonIterations = 1)
Softmax over the class ciphertexts (one per class, image i in slot i < batchSize) by repeated squaring, for logits within [-logitBound, logitBound]: q = exp(z / 2^r) (4th-order Taylor), then r times q <- q^2 / sum(q^2). Values stay within [0, 1] and nothing is decrypted. he_forward_softmax’s 1 + x + x^2 / 2 is not monotonic for x < -1 and picks wrong classes on real logits (-8 … 15 in ResNet20). One group bootstrapping of the class ciphertexts per squaring.