Evaluation
This section reports the runtime, memory and accuracy of the FHEON models.
Setup. All the runtimes and memory footprints were measured on one machine:
Intel Core Ultra 7 265K (20 threads), 31 GB of RAM;
OpenFHE 1.5.1 built with OpenMP,
-DCMAKE_BUILD_TYPE=Release;each model with its own CKKS parameters (see the example pages) and minimal key sets.
They indicate the relative cost of the models and controllers.
Timing columns.
Circuit time: homomorphic evaluation of the layers.
Bootstrap time: CKKS bootstrapping.
Total time: the full inference, including key loading, encoding and decryption.
Amortized time: the total time divided by the number of images of the batch.
Memory: peak resident memory of the process.
Single-input times are per image, averaged over the evaluated images. Batch and training times are for the whole
batch. The security column gives the parameters: performance efficient (-DSECURITY_LEVEL=PERFORMANCE, not a
standard level) or a standard level such as 128-bit (see Security Levels, Ring Dimension and Packing). The single-input results are split by primitives (standard or diagonal, see
Standard Primitives and Diagonal Primitives (FHEON-D)); the single-input models run sequentially, the batch models
and the training multithreaded. Empty cells are measurements that are not available yet.
Single-input inference
The single-input models run sequentially, one image at a time.
Standard primitives
Models of HEStandardSingleModels/ on FHEONANNStandardController (see
Standard Primitives).
Model |
Security |
Memory |
Circuit time |
Bootstrap time |
Total time |
|---|---|---|---|---|---|
BasicMLP |
Performance |
0.2 GB |
4.4 s |
0.0 s |
4.4 s |
BasicMLP |
128-bit |
3.5 GB |
73.2 s |
0.0 s |
73.2 s |
LeNet5 |
Performance |
2.8 GB |
11.6 s |
2.5 s |
14.1 s |
ResNet20Basic |
Performance |
8.8 GB |
408.8 s |
64.1 s |
473.4 s |
ResNet20DataPreprocessed |
Performance |
||||
ResNet20MultiChannels |
Performance |
8.9 GB |
353.7 s |
64.9 s |
419.1 s |
ResNet20MultiChansAndSlots |
Performance |
8.6 GB |
348.9 s |
65.5 s |
437.8 s |
ResNet20Optimized |
Performance |
8.7 GB |
367.2 s |
67.7 s |
454.3 s |
ResNet34 |
Performance |
9.5 GB |
1565.6 s |
115.8 s |
1683.3 s |
ResNet34Optimized |
Performance |
9.0 GB |
1465.3 s |
119.3 s |
1612.8 s |
VGG11 |
Performance |
16.8 GB |
1424.2 s |
52.8 s |
1486.6 s |
VGG11Optimized |
Performance |
11.9 GB |
1258.2 s |
67.9 s |
1362.1 s |
VGG16 |
Performance |
16.7 GB |
2166.7 s |
58.2 s |
2234.7 s |
VGG16Optimized |
Performance |
11.9 GB |
1897.2 s |
106.8 s |
2041.2 s |
Diagonal primitives
Models of HEDiagonalSingleModels/ on FHEONANNDiagonalController (see
Diagonal Primitives (FHEON-D)).
Model |
Security |
Memory |
Circuit time |
Bootstrap time |
Total time |
|---|---|---|---|---|---|
MLPDiagonal |
Performance |
0.1 GB |
0.12 s |
0.0 s |
0.12 s |
MLPDiagonal |
128-bit |
1.5 GB |
1.84 s |
0.0 s |
1.88 s |
LeNet5Diagonal |
Performance |
4.8 GB |
1.80 s |
0.36 s |
2.17 s |
LeNet5Diagonal |
128-bit |
12.9 GB |
9.04 s |
0.0 s |
9.07 s |
LeNet5KeyOptimizedDiagonal |
Performance |
1.9 GB |
2.04 s |
0.39 s |
4.89 s |
LeNet5KeyOptimizedDiagonal |
128-bit |
8.8 GB |
9.85 s |
0.0 s |
20.3 s |
ResNet20Diagonal |
Performance |
13.1 GB |
16.0 s |
45.3 s |
84.2 s |
ResNet20Diagonal |
128-bit |
27.7 GB |
32.4 s |
82.0 s |
164.4 s |
ResNet20OptimizedDiagonal |
Performance |
15.1 GB |
16.2 s |
44.5 s |
67.3 s |
ResNet34Diagonal |
Performance |
17.3 GB |
32.3 s |
73.2 s |
137.5 s |
VGG11Diagonal |
Performance |
17.4 GB |
14.2 s |
26.8 s |
68.9 s |
VGG16Diagonal |
Performance |
17.9 GB |
17.2 s |
38.2 s |
82.9 s |
Batch inference
The batch runs capped the workers (and the batched bootstrapping) at 4 threads (--threads 4).
Model |
Batch size |
Controller |
Security |
Memory |
Circuit time |
Bootstrap time |
Total time |
Amortized time |
|---|---|---|---|---|---|---|---|---|
BLeNet5N16 |
16 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20N16 |
16 |
Standard batch, multithreaded |
Performance |
13.0 GB |
635.1 s |
1422.6 s |
2099.5 s |
131.2 s |
BResNet20N32 |
32 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20N64 |
64 |
Standard batch, multithreaded |
Performance |
13.9 GB |
1248.7 s |
2279.3 s |
3602.9 s |
56.3 s |
BResNet20N128 |
128 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20N256 |
256 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20N512 |
512 |
Standard batch, multithreaded |
Performance |
|||||
BResNet34N16 |
16 |
Standard batch, multithreaded |
Performance |
20.2 GB |
2305.7 s |
3813.3 s |
6249.3 s |
390.6 s |
BResNet34N32 |
32 |
Standard batch, multithreaded |
Performance |
|||||
BResNet34N64 |
64 |
Standard batch, multithreaded |
Performance |
18.6 GB |
3706.7 s |
4720.0 s |
8503.9 s |
132.9 s |
BResNet34N128 |
128 |
Standard batch, multithreaded |
Performance |
|||||
BResNet34N256 |
256 |
Standard batch, multithreaded |
Performance |
|||||
BResNet34N128P2 |
128 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20N64JoinedKeys, BResNet20N256JoinedKeys, BResNet34N16JoinedKeys |
64, 256, 16 |
Standard batch, multithreaded |
Performance |
|||||
BResNet20Pipeline, BResNet34Pipeline, BVGG11Pipeline, BVGG16Pipeline |
up to 256, 1024, 4096, 4096 |
Standard batch, multithreaded |
Performance |
A later run of BResNet20N16 (16 images, 16/16 correct) took 2145 s with a peak of 11.7 GB (679 s of circuit,
1427 s of bootstrapping).
Parameters at 128-bit security
The 128-bit rows of the single-input table were measured with -DSECURITY_LEVEL=128 (build/std128): OpenFHE
picks the ring dimension from the modulus chain, the programs fill its slots and lower their scale where
Security Levels, Ring Dimension and Packing allows. Times are per image, averaged over the first MNIST or CIFAR-10 test images.
Model |
Ring |
Scale |
Digits |
Chain |
Accuracy |
|---|---|---|---|---|---|
BasicMLP |
|
31 bits |
5 |
leveled |
4/4 |
MLPDiagonal |
|
32 bits |
2 |
leveled |
4/4 |
LeNet5Diagonal |
|
26 bits |
8 |
leveled |
4/4 |
LeNet5KeyOptimizedDiagonal |
|
26 bits |
8 |
leveled |
10/10 |
ResNet20Diagonal |
|
48 bits |
4 |
bootstrapped |
9/9 |
The batch models take twice their batch at 128 bits (Batch Input Examples) and need more memory and disk for their keys than this machine has, so their rows are performance-efficient only.
Training and fine-tuning
One fine-tuning step of ResNet-20 on a batch of CIFAR-10 images (see Training and Fine-Tuning Examples), with 4 worker threads.
Model |
Batch size |
Controller |
Security |
Memory |
Circuit time |
Bootstrap time |
Total time |
Amortized time |
|---|---|---|---|---|---|---|---|---|
tuneResNet20 (forward and backward) |
16 |
Standard training, multithreaded |
Performance |
Accuracy
Plaintext accuracy is that of the trained (unencrypted) model. Encrypted accuracy is that of the FHEON model on the first test images of the dataset, decrypted after the inference. Test size is the number of images evaluated in the encrypted domain. The VGG-11 and VGG-16 CIFAR-10 accuracy figures below are reported in the FHEON paper and kept here for reference in the studio docs.
Model |
Dataset |
Plaintext accuracy |
Encrypted accuracy |
Test size |
|---|---|---|---|---|
MLP |
MNIST |
98.6% |
97.8% |
1000 |
LeNet-5 |
MNIST |
98.8% |
98.9% |
1000 |
ResNet-20 |
CIFAR-10 |
94.4% |
93.8% |
500 |
ResNet-34 |
CIFAR-100 |
76.5% |
74.4% |
210 |
VGG-11 |
CIFAR-10 |
87.7% |
86.1% |
331 |
VGG-16 |
CIFAR-10 |
88.3% |
85.8% |
120 |
Notes
The standard single-input models and the batch models were measured with their minimal key sets. The standard and diagonal pairs were run one after the other on the same images, with the same predictions in every pair.
MLPDiagonal,ResNet20OptimizedDiagonal,ResNet34Diagonal,VGG11DiagonalandVGG16Diagonalwere measured in earlier runs of the diagonal validation (2 to 10 images). A later run ofVGG11Diagonal(4 images, 4/4 correct) took 76.3 s per image (14.5 s of circuit, 27.3 s of bootstrapping, 13.1 GB).LeNet5KeyOptimizedDiagonalkeeps one key file in memory at a time: it trades time (loading the key files) for memory.ResNet20DataPreprocessedpre-encodes all its weights and needs more than the 31 GB of the machine. The batch models above 64 images,BResNet34N128P2(ring2^16) and the full training steps are meant for larger servers.