Training Backward Pass Controller

Backward pass and SGD updates of encrypted training on the standard layout. See Encrypted Training.

class FHEONTrainBackwardPassController : public FHEONTrainSupportController

Backward pass of the training pipeline, standard layout.

Gradients of the cross-entropy + softmax, fully connected, convolution (stride 1, and stride 2 through a backward dilation), shortcut, batch normalization, ReLU and pooling layers, with the SGD update of every encrypted parameter written back (optimizer_update). The backward dilation exists in three variants (backward_dilate, backward_dilate_optimized, backward_dilate_logdepth) trading operations against levels. FHEONTrainStandardBackwardPassController is the multithreaded, optimized version (same files and keys).

Subclassed by FHEONTrainStandardBackwardPassController

Public Functions

inline void setHEController(FHEONHEController *controller)
inline FHEONTrainBackwardPassController(CryptoContext<DCRTPoly> &ctx)
inline void setNumSlots(int numSlots)
void welcomeToTrainingMessage()

Displays a welcome message for the FHEON Training Controller.

vector<Ctext> softmax(vector<Ctext> &encryptedInput, int inputSize, int appxPolyDegree = 59)
Ctext cross_entropy_loss(vector<Ctext> &encryptedInputs, vector<double> &labels, int batchSize, int numClasses, int appxDegree = 3)
vector<Ctext> backward_pass_cross_entropy_and_softmax(vector<Ctext> &encryptedInputs, vector<double> &labels, int batchSize, int vectorSize, string layerName = "")
vector<Ctext> backward_pass_relu(vector<Ctext> &gradInputs, vector<int> &scalers, int appxPolyDegree, string layerName = "")
vector<Ctext> backward_pass_batch_norm(vector<Ctext> &encryptedInputs, vector<Ctext> &gradInputs, int inputSize, double learningRate, string layerName = "")

Backpropagation through batch normalization. This function computes the gradient of the loss with respect to the inputs by performing the necessary computations for batch normalization backpropagation.

Parameters:
  • gradInputs – Vector of ciphertexts representing the gradients.

  • inputSize – Size of the input vector.

  • learningRate – Learning rate for updating parameters.

  • encryptedInputs – Unused: the normalized inputs (xhat) and gamma / std are read from the caches of he_forward_batch_norm.

  • layerName – Prefix of the batch norm parameter (gamma, beta) and cache files, as in he_forward_batch_norm.

Returns:

A vector of ciphertexts containing the gradients for each input.

vector<Ctext> backward_pass_fully_connected(vector<Ctext> &gradInputs, int vectorSize, int inputSize, int outputSize, double learningRate, string layerName)
vector<Ctext> backward_pass_convolution(vector<Ctext> &gradInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int kernelWidth, double learningRate, string layerName, int stride = 1)
vector<Ctext> backward_pass_avgpool(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int poolSize)
vector<Ctext> backward_pass_avgpool_legacy(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int poolSize)
vector<Ctext> backward_pass_convolution_shortcut(vector<Ctext> &gradInputs, int batchSize, int inputWidth, int inputChannels, int outputChannels, int stride, double learningRate, string layerName)
vector<Ctext> backward_pass_global_avgpool(vector<Ctext> &gradInputs, int batchSize, int dataWidth)
vector<Ctext> backward_pass_avgpool_optimized(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int dilateFactor)
vector<Ctext> backward_pass_global_avgpool_optimized(vector<Ctext> &gradInputs, int batchSize, int dataWidth)
Ctext backward_dilate_optimized(Ctext &encryptedInput, int batchSize, int inputWidth, int dilateFactor)
vector<int> backward_dilate_optimized_indices(int batchSize, int inputWidth, int dilateFactor)
vector<Ctext> backward_pass_avgpool_logdepth(vector<Ctext> &encryptedInputs, int batchSize, int inputWidth, int dilateFactor)
vector<Ctext> backward_pass_global_avgpool_logdepth(vector<Ctext> &gradInputs, int batchSize, int dataWidth)
Ctext backward_dilate_logdepth(Ctext &encryptedInput, int batchSize, int inputWidth, int dilateFactor)
vector<int> backward_dilate_logdepth_indices(int batchSize, int inputWidth, int dilateFactor)
Ctext newtonraphson_approximate_inverse_sqrt(Ctext &encryptedInput, int inputSize, double initValue = 1.0, int appxDegree = 3)
Ctext taylors_approximation(Ctext &encryptedInput, int appxDegree)
Ctext repeat_encrypted_values(Ctext &encryptedInput, int vecSize, int batchSize)
Ctext optimizer_update(Ctext &encryptedParam, Ctext &encryptedGrad, double learningRate)
Ctext optimizer_update(Ptext &constParam, Ctext &encryptedGrad, double learningRate)
Ctext backward_dilate(Ctext &encryptedInput, int batchSize, int inputWidth, int dilateFactor)
vector<int> backward_dilate_indices(int batchSize, int inputWidth, int dilateFactor)
Ptext gen_slot0_mask(uint32_t level)
Ptext gen_prefix_mask(int prefixLen, uint32_t level)
Ptext gen_binary_run_mask(int pattern, uint32_t level)
Ptext gen_stride_mask(int period, uint32_t level)
Ctext column_scatter_log_depth(Ctext &rowInput, int inputWidth, int dilateFactor)

Public Members

int num_slots = 1 << 14
int baseIndex = 1024