> For the complete documentation index, see [llms.txt](https://docs.zama.org/concrete-ml/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zama.org/concrete-ml/0.3/advanced-topics/onnx_pipeline.md). # More about ONNX Internally, Concrete-ML uses [ONNX](https://github.com/onnx/onnx) operators as intermediate representation (or IR) for manipulating machine learning models produced through export for [PyTorch](https://github.com/pytorch/pytorch), [Hummingbird](https://github.com/microsoft/hummingbird) and [skorch](https://github.com/skorch-dev/skorch). As ONNX is becoming the standard exchange format for neural networks, this allows Concrete-ML to be flexible while also making model representation manipulation quite easy. In addition, it allows for straight-forward mapping to NumPy operators, supported by Concrete-Numpy to use the Concrete stack FHE conversion capabilities. ## Torch to NumPy conversion using ONNX The diagram below gives an overview of the steps involved in the conversion of an ONNX graph to a FHE compatible format, i.e. a format that can be compiled to FHE through Concrete-Numpy. All Concrete-ML builtin models follow the same pattern for FHE conversion: 1. The models are trained with sklearn or torch 2. All models have a torch implementation for inference. This implementation is provided either by third-party tool such as [hummingbird](/concrete-ml/0.3/advanced-topics/hummingbird_usage.md), or is implemented in Concrete-ML. 3. The torch model is exported to ONNX. For more information on the use of ONNX in Concrete-ML see [here](#torch-to-numpy-conversion-using-onnx) 4. The Concrete-ML ONNX parser checks that all the operations in the ONNX graph are supported and assigns reference numpy operations to them. This step produces a `NumpyModule` 5. Quantization is performed on the [`NumpyModule`](https://github.com/zama-ai/concrete-ml/blob/release/0.3.x/docs/_apidoc/concrete.ml.torch.html#concrete.ml.torch.numpy_module.NumpyModule), producing a [`QuantizedModule`](https://github.com/zama-ai/concrete-ml/blob/release/0.3.x/docs/_apidoc/concrete.ml.quantization.html#concrete.ml.quantization.quantized_module.QuantizedModule) . Two steps are performed: calibration and assignment of equivalent [`QuantizedOp`](https://github.com/zama-ai/concrete-ml/blob/release/0.3.x/docs/_apidoc/concrete.ml.quantization.html#concrete.ml.quantization.base_quantized_op.QuantizedOp) objects to each ONNX operation. The `QuantizedModule` class is the quantized counterpart of the `NumpyModule`. 6. Once the `QuantizedModule` is built, Concrete-Numpy is used to trace the `._forward()` function of the `QuantizedModule` Moreover, by passing a user provided `nn.Module` to step 2 of the above process, Concrete-ML supports custom user models. See the associated [FHE-friendly model documentation](/concrete-ml/0.3/deep-learning/fhe_friendly_models.md) for instructions about working with such models. ![Torch compilation flow with ONNX](https://3332742850-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FVSEDLkZUW3FweoKsRQJ2%2Fuploads%2Fgit-blob-ca80ea741405a5b27fe4a129b2e4249606a8f4a1%2Ftorch_to_numpy_with_onnx.svg?alt=media) Once an ONNX model is imported, it is converted to a `NumpyModule`, then to a `QuantizedModule` and, finally, to an FHE circuit. However, as the diagram shows, it is perfectly possible to stop at the `NumpyModule` level if you just want to run the torch model as NumPy code without doing quantization. {% hint style="info" %} Note that if you keep the obtained `NumpyModule` without quantizing it with Post Training Quantization (PTQ), it will not be convertible to FHE since the Concrete stack requires operators to use integers for computations. {% endhint %} The `NumpyModule` stores the ONNX model that it interprets. The interpreter works by going through the ONNX graph in [topological order](https://en.wikipedia.org/wiki/Topological_sorting), and storing the intermediate results as it goes. To execute a node, the interpreter feeds the required inputs - taken either from the model inputs or the intermediate results - to the NumPy implementation of each ONNX node. ## Calibration Calibration is the process of executing the `NumpyModule` with a representative set of data, in floating point. It allows to compute statistics for all the intermediate tensors used in the network to determine quantization parameters. {% hint style="info" %} Note that the `NumpyModule` interpreter currently [supports the following ONNX operators](/concrete-ml/0.3/deep-learning/onnx_support.md#supported-operators). {% endhint %} ## Quantization Quantization is the process of converting floating point weights, inputs and activations to integer, according to the quantization parameters computed during Calibration. Initializers (model trained parameters) are quantized according to `n_bits` and passed to the Post Training Quantization (PTQ) process. During the PTQ process, the ONNX model stored in the `NumpyModule` is interpreted and calibrated using `ONNX_OPS_TO_QUANTIZED_IMPL` dictionary, which maps ONNX operators (e.g. Gemm) to their quantized equivalent (e.g. QuantizedGemm). For more information on implementing these operations, please see the [FHE compatible op-graph section](/concrete-ml/0.3/advanced-topics/fhe-op-graphs.md). Quantized operators are then used to create a `QuantizedModule` that, similarly to the `NumpyModule`, runs through the operators to perform the quantized inference with integers-only operations. That `QuantizedModule` is then compilable to FHE if the intermediate values conform to the 8 bits precision limit of the Concrete stack. ## Inspecting the ONNX models In order to better understand how Concrete-ML works under the hood, it is possible to access each model in their ONNX format and then either either print it or visualize it by importing the associated file in [Netron](https://netron.app). For example, with `LogisticRegression`: ```python import onnx from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from concrete.ml.sklearn import LogisticRegression # Create the data for classification x, y = make_classification(n_samples=100, class_sep=2, n_features=4, random_state=42) # Retrieve train and test sets X_train, X_test, y_train, y_test = train_test_split( x, y, test_size=10, random_state=42 ) # Fix the number of bits to used for quantization model = LogisticRegression(n_bits=2) # Fit the model model.fit(X_train, y_train) # Access to the model onnx_model = model.onnx_model # Print the model print(onnx.helper.printable_graph(onnx_model.graph)) # Save the model onnx.save(onnx_model, "tmp.onnx") # And then visualize it with Netron ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.zama.org/concrete-ml/0.3/advanced-topics/onnx_pipeline.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.