Standard Backward Pass Controller (Multithreaded)

Multithreaded, optimized version of FHEONTrainBackwardPassController (same files and keys).

class FHEONTrainStandardBackwardPassController : public FHEONTrainBackwardPassController

Multithreaded backward pass of the training pipeline, standard layout (one channel of the batch per ciphertext).

Same primitives, arguments, cache files, parameter files and results as FHEONTrainBackwardPassController (it derives from it; the dilations, pooling variants, losses and helpers are inherited), with:

  • convolution, shortcut and fully connected layers fused per input channel: every weight is read once, used for the input gradient (old weight, as the original) and updated and written in the same worker. The original reads every convolution weight twice and holds a whole outputs x inputs matrix of weight gradients (fully connected);

  • the input-gradient products accumulated without relinearization (one relinearization per input channel instead of outputChannels * 9);

  • every EvalBootstrap on the calling thread, with setBootstrapIterations (2 by default) and setBootstrapLevelThreshold (batch norm refresh above level 10 by default, as the original);

  • the number of workers capped with setWorkerThreads. The rotations are those of the original, so the existing training key files apply unchanged.

Public Functions

inline FHEONTrainStandardBackwardPassController(CryptoContext<DCRTPoly> &ctx)
inline void setWorkerThreads(int numThreads)
inline int getWorkerThreads() const
inline void setBootstrapIterations(int iterations)
inline void setBootstrapLevelThreshold(int level)
inline void setBootstrapPrecision(int bits)
inline void setBootstrapFunction(const std::function<vector<Ctext>(vector<Ctext>&)> &function)
vector<Ctext> backward_pass_cross_entropy_and_softmax(vector<Ctext> &encryptedInputs, vector<double> &labels, int batchSize, int vectorSize, string layerName = "")

(predictions - labels) / batchSize on the first batchSize slots, see FHEONTrainBackwardPassController::backward_pass_cross_entropy_and_softmax (labels and mask encoded once per class).

vector<Ctext> backward_pass_relu(vector<Ctext> &gradInputs, vector<int> &scalers, int appxPolyDegree, string layerName = "")

dInput = dOutput * step(x), step approximated with Chebyshev on the cached input, see FHEONTrainBackwardPassController::backward_pass_relu.

vector<Ctext> backward_pass_batch_norm(vector<Ctext> &encryptedInputs, vector<Ctext> &gradInputs, int inputSize, double learningRate, string layerName = "")

Batch norm backward, see FHEONTrainBackwardPassController::backward_pass_batch_norm. The refresh bootstrappings run on this thread, the gradients of all channels in parallel, each channel updating its gamma and beta in the same worker.

vector<Ctext> backward_pass_fully_connected(vector<Ctext> &gradInputs, int vectorSize, int inputSize, int outputSize, double learningRate, string layerName)

Fully connected backward, see FHEONTrainBackwardPassController::backward_pass_fully_connected.

One worker per input neuron reads each of its weights once: the old weight goes into dInput (accumulated without relinearization), then the weight is updated with sum(dOutput * input) and written. The biases are updated per output neuron afterwards. No outputs x inputs matrix of gradients is held.

vector<Ctext> backward_pass_convolution(vector<Ctext> &gradInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int kernelWidth, double learningRate, string layerName, int stride = 1)

3x3 convolution backward, see FHEONTrainBackwardPassController::backward_pass_convolution.

The 9 rotations of every (dilated) output gradient are computed in parallel, as in the original. Then one worker per input channel builds the 9 taps of its cached input and, for every output channel and tap k, reads weight k once: dInput[in] += rotatedGradOut[oc][8 - k] * W[oc][in][k] (old weight, no relinearization) W[oc][in][k] -= lr * EvalSum(gradOut[oc] * tap[in][k]) (written right away) which is the original’s arithmetic with each weight read once instead of twice.

vector<Ctext> backward_pass_convolution_shortcut(vector<Ctext> &gradInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int stride, double learningRate, string layerName)

1x1 shortcut backward, see FHEONTrainBackwardPassController::backward_pass_convolution_shortcut: fused per input channel like backward_pass_fully_connected (each weight read once, old weight for dInput).

vector<Ctext> backward_pass_global_avgpool(vector<Ctext> &gradInputs, int batchSize, int dataWidth)

see FHEONTrainBackwardPassController::backward_pass_global_avgpool (same keys), with the worker cap

vector<Ctext> backward_pass_global_avgpool_logdepth(vector<Ctext> &gradInputs, int batchSize, int dataWidth)

see FHEONTrainBackwardPassController::backward_pass_global_avgpool_logdepth, with the worker cap

vector<Ctext> backward_pass_avgpool_logdepth(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int dilateFactor)

2x2 average pooling backward, see FHEONTrainBackwardPassController::backward_pass_avgpool_logdepth (the 3 window shifts hoisted on one precomputation).