concrete.ml.sklearn.qnn_module.md

module concrete.ml.sklearn.qnn_module

Sparse Quantized Neural Network torch module.

---

class SparseQuantNeuralNetwork

Sparse Quantized Neural Network.

This class implements an MLP that is compatible with FHE constraints. The weights and activations are quantized to low bit-width and pruning is used to ensure accumulators do not surpass an user-provided accumulator bit-width. The number of classes and number of layers are specified by the user, as well as the breadth of the network

method __init__

__init__(
    input_dim: int,
    n_layers: int,
    n_outputs: int,
    n_hidden_neurons_multiplier: int = 4,
    n_w_bits: int = 4,
    n_a_bits: int = 4,
    n_accum_bits: int = 32,
    n_prune_neurons_percentage: float = 0.0,
    activation_function: Type = <class 'torch.nn.modules.activation.ReLU'>,
    quant_narrow: bool = False,
    quant_signed: bool = True,
    power_of_two_scaling: bool = True
)

Sparse Quantized Neural Network constructor.

Args:

• input_dim (int): Number of dimensions of the input data.

• n_layers (int): Number of linear layers for this network.

• n_outputs (int): Number of output classes or regression targets.

• n_w_bits (int): Number of weight bits.

• n_a_bits (int): Number of activation and input bits.

• n_accum_bits (int): Maximal allowed bit-width of intermediate accumulators.

• n_hidden_neurons_multiplier (int): The number of neurons on the hidden will be the number of dimensions of the input multiplied by n_hidden_neurons_multiplier. Note that pruning is used to adjust the accumulator size to attempt to keep the maximum accumulator bit-width to n_accum_bits, meaning that not all hidden layer neurons will be active. The default value for n_hidden_neurons_multiplier is chosen for small dimensions of the input. Reducing this value decreases the FHE inference time considerably but also decreases the robustness and accuracy of model training.

• n_prune_neurons_percentage (float): The percentage of neurons to prune in the hidden layers. This can be used when setting n_hidden_neurons_multiplier with a high number (3-4), once good accuracy is obtained, in order to speed up the model in FHE.

• activation_function (Type): The activation function to use in the network (e.g., torch.ReLU, torch.SELU, torch.Sigmoid, ...).

• quant_narrow (bool): Whether this network should quantize the values using narrow range (e.g a 2-bits signed quantization uses [-1, 0, 1] instead of [-2, -1, 0, 1]).

• quant_signed (bool): Whether this network should quantize the values using signed integers.

• power_of_two_scaling (bool): Force quantization scales to be a power of two to enable inference speed optimizations. Defaults to True

Raises:

• ValueError: If the parameters have invalid values or the computed accumulator bit-width is zero.

---

method enable_pruning

enable_pruning() → None

Enable pruning in the network. Pruning must be made permanent to recover pruned weights.

Raises:

• ValueError: If the quantization parameters are invalid.

---

method forward

forward(x: Tensor) → Tensor

Forward pass.

Args:

• x (torch.Tensor): network input

Returns:

• x (torch.Tensor): network prediction

---

method make_pruning_permanent

make_pruning_permanent() → None

Make the learned pruning permanent in the network.

---

method max_active_neurons

max_active_neurons() → int

Compute the maximum number of active (non-zero weight) neurons.

The computation is done using the quantization parameters passed to the constructor. Warning: With the current quantization algorithm (asymmetric) the value returned by this function is not guaranteed to ensure FHE compatibility. For some weight distributions, weights that are 0 (which are pruned weights) will not be quantized to 0. Therefore the total number of active quantized neurons will not be equal to max_active_neurons.

Returns:

• int: The maximum number of active neurons.