Evaluation

This section reports the runtime, memory and accuracy of the FHEON models.

Setup. All the runtimes and memory footprints were measured on one machine:

  • Intel Core Ultra 7 265K (20 threads), 31 GB of RAM;

  • OpenFHE 1.5.1 built with OpenMP, -DCMAKE_BUILD_TYPE=Release;

  • each model with its own CKKS parameters (see the example pages) and minimal key sets.

They indicate the relative cost of the models and controllers.

Timing columns.

  • Circuit time: homomorphic evaluation of the layers.

  • Bootstrap time: CKKS bootstrapping.

  • Total time: the full inference, including key loading, encoding and decryption.

  • Amortized time: the total time divided by the number of images of the batch.

  • Memory: peak resident memory of the process.

Single-input times are per image, averaged over the evaluated images. Batch and training times are for the whole batch. The security column gives the parameters: performance efficient (-DSECURITY_LEVEL=PERFORMANCE, not a standard level) or a standard level such as 128-bit (see Security Levels, Ring Dimension and Packing). The single-input results are split by primitives (standard or diagonal, see Standard Primitives and Diagonal Primitives (FHEON-D)); the single-input models run sequentially, the batch models and the training multithreaded. Empty cells are measurements that are not available yet.

Single-input inference

The single-input models run sequentially, one image at a time.

Standard primitives

Models of HEStandardSingleModels/ on FHEONANNStandardController (see Standard Primitives).

Model

Security

Memory

Circuit time

Bootstrap time

Total time

BasicMLP

Performance

0.2 GB

4.4 s

0.0 s

4.4 s

BasicMLP

128-bit

3.5 GB

73.2 s

0.0 s

73.2 s

LeNet5

Performance

2.8 GB

11.6 s

2.5 s

14.1 s

ResNet20Basic

Performance

8.8 GB

408.8 s

64.1 s

473.4 s

ResNet20DataPreprocessed

Performance

ResNet20MultiChannels

Performance

8.9 GB

353.7 s

64.9 s

419.1 s

ResNet20MultiChansAndSlots

Performance

8.6 GB

348.9 s

65.5 s

437.8 s

ResNet20Optimized

Performance

8.7 GB

367.2 s

67.7 s

454.3 s

ResNet34

Performance

9.5 GB

1565.6 s

115.8 s

1683.3 s

ResNet34Optimized

Performance

9.0 GB

1465.3 s

119.3 s

1612.8 s

VGG11

Performance

16.8 GB

1424.2 s

52.8 s

1486.6 s

VGG11Optimized

Performance

11.9 GB

1258.2 s

67.9 s

1362.1 s

VGG16

Performance

16.7 GB

2166.7 s

58.2 s

2234.7 s

VGG16Optimized

Performance

11.9 GB

1897.2 s

106.8 s

2041.2 s

Diagonal primitives

Models of HEDiagonalSingleModels/ on FHEONANNDiagonalController (see Diagonal Primitives (FHEON-D)).

Model

Security

Memory

Circuit time

Bootstrap time

Total time

MLPDiagonal

Performance

0.1 GB

0.12 s

0.0 s

0.12 s

MLPDiagonal

128-bit

1.5 GB

1.84 s

0.0 s

1.88 s

LeNet5Diagonal

Performance

4.8 GB

1.80 s

0.36 s

2.17 s

LeNet5Diagonal

128-bit

12.9 GB

9.04 s

0.0 s

9.07 s

LeNet5KeyOptimizedDiagonal

Performance

1.9 GB

2.04 s

0.39 s

4.89 s

LeNet5KeyOptimizedDiagonal

128-bit

8.8 GB

9.85 s

0.0 s

20.3 s

ResNet20Diagonal

Performance

13.1 GB

16.0 s

45.3 s

84.2 s

ResNet20Diagonal

128-bit

27.7 GB

32.4 s

82.0 s

164.4 s

ResNet20OptimizedDiagonal

Performance

15.1 GB

16.2 s

44.5 s

67.3 s

ResNet34Diagonal

Performance

17.3 GB

32.3 s

73.2 s

137.5 s

VGG11Diagonal

Performance

17.4 GB

14.2 s

26.8 s

68.9 s

VGG16Diagonal

Performance

17.9 GB

17.2 s

38.2 s

82.9 s

Batch inference

The batch runs capped the workers (and the batched bootstrapping) at 4 threads (--threads 4).

Model

Batch size

Controller

Security

Memory

Circuit time

Bootstrap time

Total time

Amortized time

BLeNet5N16

16

Standard batch, multithreaded

Performance

BResNet20N16

16

Standard batch, multithreaded

Performance

13.0 GB

635.1 s

1422.6 s

2099.5 s

131.2 s

BResNet20N32

32

Standard batch, multithreaded

Performance

BResNet20N64

64

Standard batch, multithreaded

Performance

13.9 GB

1248.7 s

2279.3 s

3602.9 s

56.3 s

BResNet20N128

128

Standard batch, multithreaded

Performance

BResNet20N256

256

Standard batch, multithreaded

Performance

BResNet20N512

512

Standard batch, multithreaded

Performance

BResNet34N16

16

Standard batch, multithreaded

Performance

20.2 GB

2305.7 s

3813.3 s

6249.3 s

390.6 s

BResNet34N32

32

Standard batch, multithreaded

Performance

BResNet34N64

64

Standard batch, multithreaded

Performance

18.6 GB

3706.7 s

4720.0 s

8503.9 s

132.9 s

BResNet34N128

128

Standard batch, multithreaded

Performance

BResNet34N256

256

Standard batch, multithreaded

Performance

BResNet34N128P2

128

Standard batch, multithreaded

Performance

BResNet20N64JoinedKeys, BResNet20N256JoinedKeys, BResNet34N16JoinedKeys

64, 256, 16

Standard batch, multithreaded

Performance

BResNet20Pipeline, BResNet34Pipeline, BVGG11Pipeline, BVGG16Pipeline

up to 256, 1024, 4096, 4096

Standard batch, multithreaded

Performance

A later run of BResNet20N16 (16 images, 16/16 correct) took 2145 s with a peak of 11.7 GB (679 s of circuit, 1427 s of bootstrapping).

Parameters at 128-bit security

The 128-bit rows of the single-input table were measured with -DSECURITY_LEVEL=128 (build/std128): OpenFHE picks the ring dimension from the modulus chain, the programs fill its slots and lower their scale where Security Levels, Ring Dimension and Packing allows. Times are per image, averaged over the first MNIST or CIFAR-10 test images.

Model

Ring

Scale

Digits

Chain

Accuracy

BasicMLP

2^15

31 bits

5

leveled

4/4

MLPDiagonal

2^15

32 bits

2

leveled

4/4

LeNet5Diagonal

2^15

26 bits

8

leveled

4/4

LeNet5KeyOptimizedDiagonal

2^15

26 bits

8

leveled

10/10

ResNet20Diagonal

2^16

48 bits

4

bootstrapped

9/9

The batch models take twice their batch at 128 bits (Batch Input Examples) and need more memory and disk for their keys than this machine has, so their rows are performance-efficient only.

Training and fine-tuning

One fine-tuning step of ResNet-20 on a batch of CIFAR-10 images (see Training and Fine-Tuning Examples), with 4 worker threads.

Model

Batch size

Controller

Security

Memory

Circuit time

Bootstrap time

Total time

Amortized time

tuneResNet20 (forward and backward)

16

Standard training, multithreaded

Performance

Accuracy

Plaintext accuracy is that of the trained (unencrypted) model. Encrypted accuracy is that of the FHEON model on the first test images of the dataset, decrypted after the inference. Test size is the number of images evaluated in the encrypted domain. The VGG-11 and VGG-16 CIFAR-10 accuracy figures below are reported in the FHEON paper and kept here for reference in the studio docs.

Model

Dataset

Plaintext accuracy

Encrypted accuracy

Test size

MLP

MNIST

98.6%

97.8%

1000

LeNet-5

MNIST

98.8%

98.9%

1000

ResNet-20

CIFAR-10

94.4%

93.8%

500

ResNet-34

CIFAR-100

76.5%

74.4%

210

VGG-11

CIFAR-10

87.7%

86.1%

331

VGG-16

CIFAR-10

88.3%

85.8%

120

Notes

  • The standard single-input models and the batch models were measured with their minimal key sets. The standard and diagonal pairs were run one after the other on the same images, with the same predictions in every pair.

  • MLPDiagonal, ResNet20OptimizedDiagonal, ResNet34Diagonal, VGG11Diagonal and VGG16Diagonal were measured in earlier runs of the diagonal validation (2 to 10 images). A later run of VGG11Diagonal (4 images, 4/4 correct) took 76.3 s per image (14.5 s of circuit, 27.3 s of bootstrapping, 13.1 GB).

  • LeNet5KeyOptimizedDiagonal keeps one key file in memory at a time: it trades time (loading the key files) for memory.

  • ResNet20DataPreprocessed pre-encodes all its weights and needs more than the 31 GB of the machine. The batch models above 64 images, BResNet34N128P2 (ring 2^16) and the full training steps are meant for larger servers.