> For the complete documentation index, see [llms.txt](https://docs.zama.org/tfhe-rs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zama.org/tfhe-rs/tfhe-rs/0.7-1/guides/run_on_gpu.md).

# GPU acceleration

This guide explains how to update your existing program to leverage GPU acceleration, or to start a new program using GPU.

**TFHE-rs** now supports a GPU backend with CUDA implementation, enabling integer arithmetic operations on encrypted data.

## Prerequisites

* Cuda version >= 10
* Compute Capability >= 3.0
* [gcc](https://gcc.gnu.org/) >= 8.0 - check this [page](https://gist.github.com/ax3l/9489132) for more details about nvcc/gcc compatible versions
* [cmake](https://cmake.org/) >= 3.24
* Rust version - check this [page](/tfhe-rs/tfhe-rs/0.7-1/guides/rust_configuration.md)

## Importing to your project

To use the **TFHE-rs** GPU backend in your project, add the following dependency in your `Cargo.toml`.

If you are using an `x86` machine:

```toml
tfhe = { version = "0.7.5", features = [ "boolean", "shortint", "integer", "x86_64-unix", "gpu" ] }
```

If you are using an `ARM` machine:

```toml
tfhe = { version = "0.7.5", features = [ "boolean", "shortint", "integer", "aarch64-unix", "gpu" ] }
```

{% hint style="success" %}
For optimal performance when using **TFHE-rs**, run your code in release mode with the `--release` flag.
{% endhint %}

### Supported platforms

**TFHE-rs** GPU backend is supported on Linux (x86, aarch64).

| OS      | x86           | aarch64          |
| ------- | ------------- | ---------------- |
| Linux   | `x86_64-unix` | `aarch64-unix`\* |
| macOS   | Unsupported   | Unsupported\*    |
| Windows | Unsupported   | Unsupported      |

## A first example

### Configuring and creating keys.

Comparing to the [CPU example](/tfhe-rs/tfhe-rs/0.7-1/get-started/quick_start.md), GPU set up differs in the key creation, as detailed [here](#Setting-the-keys)

Here is a full example (combining the client and server parts):

```rust
use tfhe::{ConfigBuilder, set_server_key, FheUint8, ClientKey, CompressedServerKey};
use tfhe::prelude::*;

fn main() {

    let config = ConfigBuilder::default().build();

    let client_key= ClientKey::generate(config);
    let compressed_server_key = CompressedServerKey::new(&client_key);

    let gpu_key = compressed_server_key.decompress_to_gpu();

    let clear_a = 27u8;
    let clear_b = 128u8;

    let a = FheUint8::encrypt(clear_a, &client_key);
    let b = FheUint8::encrypt(clear_b, &client_key);

    //Server-side

    set_server_key(gpu_key);
    let result = a + b;

    //Client-side
    let decrypted_result: u8 = result.decrypt(&client_key);

    let clear_result = clear_a + clear_b;

    assert_eq!(decrypted_result, clear_result);
}
```

### Setting the keys

The configuration of the key is different from the CPU. More precisely, if both client and server keys are still generated by the client (which is assumed to run on a CPU), the server key has then to be decompressed by the server to be converted into the right format. To do so, the server should run this function: `decompressed_to_gpu()`.

Once decompressed, the operations between CPU and GPU are identical.

### Encryption

On the client-side, the method to encrypt the data is exactly the same than the CPU one, as shown in the following example:

```Rust
    let clear_a = 27u8;
    let clear_b = 128u8;
    
    let a = FheUint8::encrypt(clear_a, &client_key);
    let b = FheUint8::encrypt(clear_b, &client_key);
```

### Computation

The server first need to set up its keys with `set_server_key(gpu_key)`.

Then, homomorphic computations are performed using the same approach as the [CPU operations](/tfhe-rs/tfhe-rs/0.7-1/get-started/operations.md).

```Rust
    //Server-side
    set_server_key(gpu_key);
    let result = a + b;

    //Client-side
    let decrypted_result: u8 = result.decrypt(&client_key);

    let clear_result = clear_a + clear_b;

    assert_eq!(decrypted_result, clear_result);
```

### Decryption

Finally, the client decrypts the results using:

```Rust
    let decrypted_result: u8 = result.decrypt(&client_key);
```

### Improving performance

**TFHE-rs** allows to leverage the high number of threads given by a GPU. To maximize the number of GPU threads, update your configuration accordingly:

```Rust
let config = ConfigBuilder::with_custom_parameters(PARAM_GPU_MULTI_BIT_MESSAGE_2_CARRY_2_GROUP_3_KS_PBS, None).build();
```

Here's the complete example:

```rust
use tfhe::{ConfigBuilder, set_server_key, FheUint8, ClientKey, CompressedServerKey};
use tfhe::prelude::*;
use tfhe::shortint::parameters::PARAM_GPU_MULTI_BIT_MESSAGE_2_CARRY_2_GROUP_3_KS_PBS;

fn main() {

    let config = ConfigBuilder::with_custom_parameters(PARAM_GPU_MULTI_BIT_MESSAGE_2_CARRY_2_GROUP_3_KS_PBS, None).build();

    let client_key= ClientKey::generate(config);
    let compressed_server_key = CompressedServerKey::new(&client_key);

    let gpu_key = compressed_server_key.decompress_to_gpu();

    let clear_a = 27u8;
    let clear_b = 128u8;

    let a = FheUint8::encrypt(clear_a, &client_key);
    let b = FheUint8::encrypt(clear_b, &client_key);

    //Server-side

    set_server_key(gpu_key);
    let result = a + b;

    //Client-side
    let decrypted_result: u8 = result.decrypt(&client_key);

    let clear_result = clear_a + clear_b;

    assert_eq!(decrypted_result, clear_result);
}
```

## List of available operations

The GPU backend includes the following operations for both signed and unsigned encrypted integers:

| name                  | symbol         | `Enc`/`Enc`                | `Enc`/ `Int`               |
| --------------------- | -------------- | -------------------------- | -------------------------- |
| Neg                   | `-`            | :heavy\_check\_mark:       | N/A                        |
| Add                   | `+`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Sub                   | `-`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Mul                   | `*`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Div                   | `/`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Rem                   | `%`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Not                   | `!`            | :heavy\_check\_mark:       | N/A                        |
| BitAnd                | `&`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| BitOr                 | `\|`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| BitXor                | `^`            | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Shr                   | `>>`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Shl                   | `<<`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Rotate right          | `rotate_right` | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Rotate left           | `rotate_left`  | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Min                   | `min`          | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Max                   | `max`          | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Greater than          | `gt`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Greater or equal than | `ge`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Lower than            | `lt`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Lower or equal than   | `le`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Equal                 | `eq`           | :heavy\_check\_mark:       | :heavy\_check\_mark:       |
| Cast (into dest type) | `cast_into`    | :heavy\_multiplication\_x: | N/A                        |
| Cast (from src type)  | `cast_from`    | :heavy\_multiplication\_x: | N/A                        |
| Ternary operator      | `select`       | :heavy\_check\_mark:       | :heavy\_multiplication\_x: |

{% hint style="info" %}
All operations follow the same syntax than the one described in [here](/tfhe-rs/tfhe-rs/0.7-1/get-started/operations.md).
{% endhint %}

## Multi-GPU support

TFHE-rs supports platforms with multiple GPUs with some restrictions at the moment: the platform should have NVLink support, and only GPUs that have peer access to GPU 0 via NVLink will be used for the computation. Depending on the platform, this can restrict the number of GPUs used to perform the computation.

There is **nothing to change in the code to execute on multiple GPUs**, when they are available and have peer access to GPU 0 via NVLink. To keep the API as user-friendly as possible, the configuration is automatically set, i.e., the user has no fine-grained control over the number of GPUs to be used.

## Benchmarks

All GPU benchmarks presented here were obtained on H100 GPUs, and rely on the multithreaded PBS algorithm. The cryptographic parameters `PARAM_GPU_MULTI_BIT_MESSAGE_2_CARRY_2_GROUP_3_KS_PBS` were used.

### 1xH100

Below come the results for the execution on a single H100. The following table shows the performance when the inputs of the benchmarked operation are encrypted:

| Operation \ Size                                       | `FheUint8` | `FheUint16` | `FheUint32` | `FheUint64` | `FheUint128` | `FheUint256` |
| ------------------------------------------------------ | ---------- | ----------- | ----------- | ----------- | ------------ | ------------ |
| Negation (`-`)                                         | 18.6 ms    | 24.9 ms     | 34.9 ms     | 52.4 ms     | 101 ms       | 197 ms       |
| Add / Sub (`+`,`-`)                                    | 18.7 ms    | 25.0 ms     | 35.0 ms     | 52.4 ms     | 101 ms       | 197 ms       |
| Mul (`x`)                                              | 35.0 ms    | 59.7 ms     | 124 ms      | 378 ms      | 1.31 s       | 5.01 s       |
| Equal / Not Equal (`eq`, `ne`)                         | 10.5 ms    | 11.1 ms     | 17.2 ms     | 19.5 ms     | 27.9 ms      | 45.2 ms      |
| Comparisons (`ge`, `gt`, `le`, `lt`)                   | 19.8 ms    | 25.0 ms     | 31.3 ms     | 40.2 ms     | 53.2 ms      | 85.2 ms      |
| Max / Min (`max`,`min`)                                | 30.2 ms    | 37.1 ms     | 46.6 ms     | 61.4 ms     | 91.8 ms      | 154 ms       |
| Bitwise operations (`&`, `\|`, `^`)                    | 4.83 ms    | 5.3 ms      | 6.36 ms     | 8.26 ms     | 15.3 ms      | 25.4 ms      |
| Div / Rem (`/`, `%`)                                   | 221 ms     | 528 ms      | 1.31 s      | 3.6 s       | 11.0 s       | 40.0 s       |
| Left / Right Shifts (`<<`, `>>`)                       | 30.4 ms    | 41.4 ms     | 60.0 ms     | 119 ms      | 221 ms       | 435 ms       |
| Left / Right Rotations (`left_rotate`, `right_rotate`) | 30.4 ms    | 41.4 ms     | 60.1 ms     | 119 ms      | 221 ms       | 435 ms       |

The following table shows the performance when the left input of the benchmarked operation is encrypted and the other is a clear scalar of the same size:

| Operation \ Size                                       | `FheUint8` | `FheUint16` | `FheUint32` | `FheUint64` | `FheUint128` | `FheUint256` |
| ------------------------------------------------------ | ---------- | ----------- | ----------- | ----------- | ------------ | ------------ |
| Add / Sub (`+`,`-`)                                    | 19.0 ms    | 25.0 ms     | 35.0 ms     | 52.4 ms     | 101 ms       | 197 ms       |
| Mul (`x`)                                              | 28.1 ms    | 43.9 ms     | 75.4 ms     | 177 ms      | 544 ms       | 1.92 s       |
| Equal / Not Equal (`eq`, `ne`)                         | 11.5 ms    | 11.9 ms     | 12.5 ms     | 18.9 ms     | 21.7 ms      | 30.6 ms      |
| Comparisons (`ge`, `gt`, `le`, `lt`)                   | 12.5 ms    | 17.4 ms     | 22.7 ms     | 29.9 ms     | 39.1 ms      | 57.2 ms      |
| Max / Min (`max`,`min`)                                | 22.5 ms    | 28.9 ms     | 37.4 ms     | 50.6 ms     | 77.4 ms      | 126 ms       |
| Bitwise operations (`&`, `\|`, `^`)                    | 4.92 ms    | 5.51 ms     | 6.47 ms     | 8.37 ms     | 15.5 ms      | 25.6 ms      |
| Div (`/`)                                              | 46.8 ms    | 70.0 ms     | 138 ms      | 354 ms      | 1.10 s       | 3.83 s       |
| Rem (`%`)                                              | 90.0 ms    | 140 ms      | 250 ms      | 592 ms      | 1.75 s       | 6.06 s       |
| Left / Right Shifts (`<<`, `>>`)                       | 4.82 ms    | 5.36 ms     | 6.38 ms     | 8.26 ms     | 15.3 ms      | 25.4 ms      |
| Left / Right Rotations (`left_rotate`, `right_rotate`) | 4.81 ms    | 5.36 ms     | 6.30 ms     | 8.19 ms     | 15.3 ms      | 25.3 ms      |

### 2xH100

Below come the results for the execution on two H100's. The following table shows the performance when the inputs of the benchmarked operation are encrypted:

| Operation \ Size                                       | `FheUint8` | `FheUint16` | `FheUint32` | `FheUint64` | `FheUint128` | `FheUint256` |
| ------------------------------------------------------ | ---------- | ----------- | ----------- | ----------- | ------------ | ------------ |
| Negation (`-`)                                         | 16.1 ms    | 20.3 ms     | 27.7 ms     | 38.2 ms     | 54.7 ms      | 83.0 ms      |
| Add / Sub (`+`,`-`)                                    | 16.1 ms    | 20.4 ms     | 27.8 ms     | 38.3 ms     | 54.9 ms      | 83.2 ms      |
| Mul (`x`)                                              | 31.0 ms    | 49.6 ms     | 92.4 ms     | 267 ms      | 892 ms       | 3.45 s       |
| Equal / Not Equal (`eq`, `ne`)                         | 11.2 ms    | 12.9 ms     | 20.4 ms     | 27.3 ms     | 38.8 ms      | 67.0 ms      |
| Max / Min (`max`,`min`)                                | 53.4 ms    | 59.3 ms     | 70.4 ms     | 89.6 ms     | 120 ms       | 177 ms       |
| Bitwise operations (`&`, `\|`, `^`)                    | 4.16 ms    | 4.62 ms     | 5.61 ms     | 7.52 ms     | 10.2 ms      | 15.7 ms      |
| Div / Rem (`/`, `%`)                                   | 299 ms     | 595 ms      | 1.36 s      | 3.12 s      | 7.8 s        | 21.1 s       |
| Left / Right Shifts (`<<`, `>>`)                       | 26.9 ms    | 34.5 ms     | 48.7 ms     | 70.2 ms     | 108 ms       | 220 ms       |
| Left / Right Rotations (`left_rotate`, `right_rotate`) | 26.8 ms    | 34.5 ms     | 48.7 ms     | 70.1 ms     | 108 ms       | 220 ms       |

The following table shows the performance when the left input of the benchmarked operation is encrypted and the other is a clear scalar of the same size:

| Operation \ Size                                       | `FheUint8` | `FheUint16` | `FheUint32` | `FheUint64` | `FheUint128` | `FheUint256` |
| ------------------------------------------------------ | ---------- | ----------- | ----------- | ----------- | ------------ | ------------ |
| Add / Sub (`+`,`-`)                                    | 16.4 ms    | 20.5 ms     | 28.0 ms     | 38.4 ms     | 54.9 ms      | 83.1 ms      |
| Mul (`x`)                                              | 25.3 ms    | 36.8 ms     | 62.0 ms     | 130 ms      | 377 ms       | 1.35 s       |
| Equal / Not Equal (`eq`, `ne`)                         | 36.4 ms    | 36.5 ms     | 39.3 ms     | 47.1 ms     | 58.0 ms      | 78.0 ms      |
| Max / Min (`max`,`min`)                                | 53.6 ms    | 60.8 ms     | 71.9 ms     | 89.4 ms     | 119 ms       | 173 ms       |
| Bitwise operations (`&`, `\|`, `^`)                    | 4.33 ms    | 4.76 ms     | 6.4 ms      | 7.65 ms     | 10.4 ms      | 15.7 ms      |
| Div (`/`)                                              | 40.9 ms    | 59.7 ms     | 109.0 ms    | 248.5 ms    | 806.1 ms     | 2.9 s        |
| Rem (`%`)                                              | 80.6 ms    | 116.1 ms    | 199.9 ms    | 412.9 ms    | 1.2 s        | 4.3 s        |
| Left / Right Shifts (`<<`, `>>`)                       | 4.15 ms    | 4.57 ms     | 6.19 ms     | 7.48 ms     | 10.3 ms      | 15.7 ms      |
| Left / Right Rotations (`left_rotate`, `right_rotate`) | 4.15 ms    | 4.57 ms     | 6.18 ms     | 7.46 ms     | 10.2 ms      | 15.6 ms      |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.zama.org/tfhe-rs/tfhe-rs/0.7-1/guides/run_on_gpu.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.