Benchmarking

This document gives an overview of the benchmarking infrastructure of Concrete.

Concrete Python

Concrete Python uses progress-tracker-python to do benchmarks. Please refer to its README to learn how it works.

How to run all benchmarks?

Use the makefile target:

make benchmark

Note that this command removes the previous benchmark results before doing the benchmark.

How to run a single benchmark?

Since the full benchmark suite takes a long time to run, it's not recommended for development. Instead, use the following command to run just a single benchmark.

TARGET=foo make benchmark-target

This command would only run the benchmarks defined in benchmarks/foo.py. It also retains the previous runs, so it can be run back to back to collect data from multiple benchmarks.

How to add new benchmarks?

Simply add a new Python script in benchmarks directory and write your logic.

The recommended file structure is as follows:

# import progress tracker
import py_progress_tracker as progress

# import any other dependencies
from concrete import fhe

# create a list of targets to benchmark
targets = [
    {
        "id": (
            f"name-of-the-benchmark :: "
            f"parameter1 = {foo} | parameter2 = {bar}"
        ),
        "name": (
            f"Name of the benchmark with parameter1 of {foo} and parameter2 of {bar}"
        ),
        "parameters": {
            "parameter1": foo,
            "parameter2": bar,
        },
    }
]

# write the benchmark logic
@progress.track(targets)
def main(parameter1, parameter2):
    ...

    # to track timings
    with progress.measure(id="some-metric-ms", label="Some metric (ms)"):
        # execution time of this block will be measured
        ...

    ...

    # to track values
    progress.measure(id="another-metric", label="Another metric", value=some_metric)

    ...

Feel free to check benchmarks/primitive.py to see this structure in action.