Linear models

This page explains Concrete ML linear models for both classification and regression. These models are based on scikit-learn linear models.

Supported models for encrypted inference

The following models are supported for training on clear data and predicting on encrypted data. Their API is similar to the one of scikit-learn. These models are also compatible with some of scikit-learn's main workflows, such as Pipeline() and GridSearch().

Concrete ML	scikit-learn
LinearRegression	LinearRegression
LogisticRegression	LogisticRegression
LinearSVC	LinearSVC
LinearSVR	LinearSVR
PoissonRegressor	PoissonRegressor
TweedieRegressor	TweedieRegressor
GammaRegressor	GammaRegressor
Lasso	Lasso
Ridge	Ridge
ElasticNet	ElasticNet
SGDRegressor	SGDRegressor

Supported models for encrypted training

In addition to predicting on encrypted data, the following models support training on encrypted data.

| SGDClassifier | SGDClassifier |

Ciphertext format compatibility

These models only support Concrete ciphertexts. See the ciphertexts format documentation for more details.

Quantization parameters

The n_bits parameter controls the bit-width of the inputs and weights of the linear models. Linear models do not use table lookups and thus allows weight and inputs to be high precision integers.

For models with input dimensions up to 300, the parameter n_bits can be set to 8 or more. When the input dimensions are larger, n_bits must be reduced to 6-7. In many cases, quantized models can preserve all performance metrics compared to the non-quantized float models from scikit-learn when n_bits is down to 6. You should validate accuracy on held-out test sets and adjust n_bits accordingly.

For optimal results, you can use standard or min-max normalization to achieve a similar distribution of individual features. When there are many one-hot features, consider Principal Component Analysis as a pre-processing stage.

For a more detailed comparison of the impact of such pre-processing, please refer to the logistic regression notebook.

Pre-trained models

You can convert an already trained scikit-learn linear model to a Concrete ML one by using the from_sklearn_model method. See the following example.

Example

The following example shows how to train a LogisticRegression model on a simple data-set and then use FHE to perform inference on encrypted data. You can find a more complete example in the LogisticRegression notebook.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from concrete.ml.sklearn import LogisticRegression

# Create the data for classification:
X, y = make_classification(
    n_features=30,
    n_redundant=0,
    n_informative=2,
    random_state=2,
    n_clusters_per_class=1,
    n_samples=250,
)

# Retrieve train and test sets:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

# Instantiate the model:
model = LogisticRegression(n_bits=8)

# Fit the model:
model.fit(X_train, y_train)

# Evaluate the model on the test set in clear:
y_pred_clear = model.predict(X_test)

# Compile the model:
model.compile(X_train)

# Perform the inference in FHE:
y_pred_fhe = model.predict(X_test, fhe="execute")

# Assert that FHE predictions are the same as the clear predictions:
print(
    f"{(y_pred_fhe == y_pred_clear).sum()} examples over {len(y_pred_fhe)} "
    "have an FHE inference equal to the clear inference."
)

# Output:
#  100 examples over 100 have an FHE inference equal to the clear inference

Model accuracy

The figure below compares the decision boundary of the FHE classifier and a scikit-learn model executed in clear. You can find the complete code in the LogisticRegression notebook.

The overall accuracy scores are identical (93%) between the scikit-learn model (executed in the clear) and the Concrete ML one (executed in FHE). In fact, quantization has little impact on the decision boundaries, as linear models can use large precision numbers when quantizing inputs and weights in Concrete ML. Additionally, as the linear models do not use Programmable Boostrapping, the FHE computations are always exact, irrespective of the PBS error tolerance setting. This ensures that the FHE predictions are always identical to the quantized clear ones.

Sklearn model decision boundaries

FHE model decision boundaries

Loading a pre-trained model

An alternative to the example above is to train a scikit-learn model in a separate step and then to convert it to Concrete ML.

from sklearn.linear_model import LogisticRegression as SKlearnLogisticRegression

# Instantiate the model:
model = SKlearnLogisticRegression()

# Fit the model:
model.fit(X_train, y_train)

cml_model = LogisticRegression.from_sklearn_model(model, X_train, n_bits=8)

# Compile the model:
cml_model.compile(X_train)

# Perform the inference in FHE:
y_pred_fhe = cml_model.predict(X_test, fhe="execute")

Zama 5-Question Developer Survey

We want to hear from you! Take 1 minute to share your thoughts and help us enhance our documentation and libraries. 👉 Click here to participate.