UtilsBatchData
Loading and reshaping the weights, biases and inputs of the batch models.
Manage weights and baises for batch neural networks.
This file provides functions to handle data preparation tasks, including generating random datasets, reading data from files for different datasets, and organizing it for use in HE-friendly neural networks.
-
namespace utilsbatchdata
Functions
-
static inline int num_batches(int numImages, int batchSize)
number of batches of batchSize images needed for numImages images
-
static inline vector<vector<double>> batch_images(const vector<vector<double>> &imagesData, int first, int batchSize, int endImage)
The batchSize images of a batch: images first, first + 1, … of imagesData, blank from endImage on.
- Parameters:
imagesData – Images read from the dataset (at least endImage of them).
first – Index in imagesData of the first image of the batch.
batchSize – Images per batch.
endImage – One past the last image to evaluate: the positions from there on are blank.
-
static inline vector<int> report_batch_predictions(int firstIndex, int numImages, int numClasses, const std::function<double(int, int)> &logitAt, const string &predictions_file)
Report the predictions of the real images of a batch.
For i < numImages, the label of image firstIndex + i (dataset index, from 0) is the argmax of logitAt(i, k), k < numClasses. Prints the lines of read_batch_inferenced_label (“Batch <index> -> Predicted Label: …” and the logits at verbose level 2), records the accuracy against the true labels and appends the labels to predictions_file. Blank padding images are never passed here.
- Returns:
The predicted labels.
-
static inline vector<vector<double>> loadCSV(const string &fileName)
Load numeric data from a CSV file.
Reads a CSV file and converts each value into double. Invalid values are replaced with 0.0.
- Parameters:
fileName – Path to the CSV file.
- Returns:
2D vector of doubles with CSV contents.
-
static inline vector<double> flatten_csv_data(const vector<vector<double>> &data)
Flatten a 2D vector into a 1D vector.
- Parameters:
data – 2D vector to flatten.
- Returns:
1D vector containing all elements in row-major order.
-
static inline vector<vector<double>> load_batch_bias(const string &fileName, int outputChannels, int batchSize, int inputSize)
Load and replicate bias values for batched processing.
Reads the first row of a CSV file as bias values and repeats each bias across all input positions for each batch. The final shape is [outputChannel][batch * inputSize], ready to be used in batched FHE operations.
- Parameters:
fileName – Path to the CSV file containing bias values.
outputChannels – Number of output channels.
batchSize – Number of batches.
inputSize – Number of input positions per batch (input channels * feature map size).
- Throws:
std::runtime_error – if the CSV is empty or does not contain enough bias values.
- Returns:
std::vector<std::vector<double>> 2D vector of shape [outputChannel][batchSize * inputSize].
-
static inline vector<vector<vector<vector<vector<double>>>>> load_batch_weights(const string &fileName, int outputChannels, int inputChannels, int batchSize, int rowsWidth, int imgCols)
Load and reshape convolution kernel weights from a CSV file for batched processing.
This function reads kernel weights from a CSV file and organizes them into a 5D vector suitable for homomorphic batched convolution operations. The weights are organized as [outputChannel][inputChannel][batch][rows][cols]. For each output channel and input channel, the kernel values are repeated across the batch dimension.
Note
For each output channel, the function reads input channel weights from the CSV exactly once and repeats them across the batch dimension. This layout ensures compatibility with batched homomorphic convolution operations.
- Parameters:
fileName – Path to the CSV file containing the kernel weights. The CSV should contain all weights flattened in row-major order.
outputChannels – Number of output channels (kernels).
inputChannels – Number of input channels.
batchSize – Number of batches; each input channel kernel is repeated this many times.
rowsWidth – Number of rows in each kernel.
imgCols – Number of columns in each kernel.
- Throws:
std::runtime_error – if the CSV does not contain enough weights to fill the requested shape.
- Returns:
std::vector<std::vector<std::vector<std::vector<std::vector<double>>>>> 5D vector of weights with shape [outputChannel][inputChannel][batch][rows][cols].
-
static inline vector<vector<vector<double>>> load_shortcut_batch_weights(string fileName, int batchSize, int outputChannels, int inputChannels, int inputSize)
Load shortcut batch weights and expand them for batching.
Reads weights from a CSV file, then for each weight value corresponding to [outputChannel][inputChannel], repeats it batchSize * inputSize times. This produces a structure suitable for batched homomorphic operations.
- Parameters:
fileName – Path to the CSV file containing weights.
batchSize – Number of batches.
outputChannels – Number of output channels.
inputChannels – Number of input channels.
inputSize – Size of each input vector.
- Returns:
3D vector of size [outputChannels][inputChannels][batchSize*inputSize].
-
static inline vector<vector<double>> load_batch_fc_weights(const string &fileName, int outputChannels, int batchSize, int inputChannels)
Load and reshape fully connected layer weights for batched processing.
Reads a CSV file containing FC layer weights and organizes them into a 2D vector. Each output channel contains a single vector where each input channel weight is repeated
batchSizetimes consecutively.- Parameters:
fileName – Path to the CSV file containing weights (flattened row-major).
outputChannels – Number of output neurons (output channels).
batchSize – Number of batches; each input channel weight is repeated this many times.
inputChannels – Number of input features (input channels).
- Throws:
std::runtime_error – if CSV is empty or does not contain enough weights.
- Returns:
std::vector<std::vector<double>> 2D vector of shape [outputChannel][inputChannel * batchSize].
-
static inline vector<double> load_batch_fc_bias(const string &fileName, int outputChannels, int batchSize)
Load and expand fully connected (FC) layer biases for batched inputs.
Reads a CSV containing bias values (one per output channel), and expands each bias across the batch dimension so it can be added after matrix multiplication.
- Parameters:
fileName – Path to the CSV file containing bias values (1 row, outputChannels cols).
outputChannels – Number of output channels (neurons).
batchSize – Number of inputs in the batch.
- Returns:
Flattened vector<double> of size batchSize * outputChannels, ordered as [batch0: all biases][batch1: all biases]…
-
static inline vector<vector<double>> convert_inputData(const vector<vector<double>> &inputDatas, int batchSize, int inputChannels, int inputSize)
Reshape batched input data into channel-major format.
Converts input data from [batch][inputChannels * inputSize] format into [inputChannels][batchSize * inputSize], where each channel contains all batches concatenated.
- Parameters:
inputDatas – Input data shaped as [batchSize][inputChannels * inputSize].
batchSize – Number of batches.
inputChannels – Number of input channels.
inputSize – Size of one input per channel (e.g., H*W).
- Returns:
Reshaped data as [inputChannels][batchSize * inputSize].
-
static inline vector<int> limit_to_physical_cores()
-
static inline int num_batches(int numImages, int batchSize)