Benchmarks

Comparison with other packages

We provide standalone benchmark scripts comparing monoprop against other open-source Majorana Propagation and Pauli propagation engines. The benchmark suite includes memory and time performance metrics, providing insights into the efficiency of different propagation engines.

You can find the benchmark scripts in the benches/third_party directory. The scripts are designed to be run in a controlled environment, ensuring fair comparisons between different engines.

Majorana Propagation

The chosen problem is the 1D Hubbard model, tracking the runtime, expectation value, and operator size at each Trotter step.

Number of terms, final overlap, runtime, and memory vs. circuit layers for the 1D Fermi-Hubbard model, comparing monoprop against MajoranaPropagation.jl. See the section below for details on the setup and how to reproduce these results.

SPECS

The benchmark is run on Leonardo (CINECA), in a single node with the following specifications:

• Cores: 8 Intel Xeon Platinum 8358 CPU, 2.60GHz (Ice Lake)
• RAM: 100 GB

The end-to-end workflow is:

1. Set up the Python environment.
2. Install Julia and MajoranaPropagation.jl.
3. Run the benchmarks to produce the JSONL result files.
4. Plot the results to produce majorana_results.png.

1. Set up the Python environment

# Navigate to this directory
cd benches/third_party

# Create the venv and install all Python dependencies (including monoprop itself, in editable mode)
uv sync

This installs monoprop and numpy.

2. Install Julia and MajoranaPropagation.jl

You can find the installation instructions for Julia and MajoranaPropagation.jl in the official documentation.

# Install juliaup and the latest stable Julia release
curl -fsSL https://install.julialang.org | sh

Once Julia has been installed, you can install and instantiate the required packages for the benchmark:

julia --project=@. -e 'using Pkg; Pkg.instantiate()'

Then, you can precompile the MajoranaPropagation.jl package to ensure that it is ready for use:

julia --project=@. -e 'using Pkg; Pkg.precompile()'

3. Run the benchmarks

You can run the benchmarks using the provided bash script for both Python and Julia engines. The script will execute the benchmarks and generate the results in two separate JSON files: julia_hubbard1d_benchmark_results.jsonl for the Julia engine and monoprop_hubbard1d_benchmark_results.jsonl for the Python engine.

# Run the benchmarks for both Python and Julia engines
bash run_benchmarks.sh

Each line of both JSONL files is one benchmark run (one system size / circuit depth pair), with:

• n_spinful_sites: number of spinful sites in the 1D Hubbard chain.
• n_layers: number of Trotter layers simulated.
• num_terms: number of Majorana terms kept in the operator at the final layer.
• final_overlap: expectation value at the final layer.
• runtime_seconds: wall-clock time to run all n_layers layers.
• memory_MB: operator memory footprint at the final layer, in megabytes.

The Python results additionally record num_threads, the value of the monoprop_NUM_THREADS environment variable used for the run.

4. Plot the results

uv run python plot_results.py

This reads the json file results for each simulation and produces a single image, majorana_results.png, combining four linear scale panels (number of terms, final overlap, runtime, and memory) in a 2x2 grid.

Results

Number of terms, final-state overlap, runtime, and memory footprint versus circuit depth for the 1D Hubbard model, comparing monoprop against MajoranaPropagation.jl across system sizes n=20, 40, and 60. Both engines produce matching term counts and overlaps, confirming correctness, while monoprop runs faster and uses less memory than the Julia engine as depth and system size grow.

Pauli Propagation

The chosen problem is the Trotterized time evolution of a 2D transverse-field Ising model (TFIM), tracking the runtime, expectation value, and operator size at each Trotter step.

Runtime and memory usage per Trotter step for the 2D TFIM problem, comparing monoprop against other Pauli propagation engines. See the Pauli Propagation section below for details on the setup and how to reproduce these results.

SPECS

• CPU: AMD Ryzen Threadripper PRO 7955WX 16-Cores
• RAM: 128 GB in dual channel
• GPU: Nvidia RTX 5090 with 32 GB VRAM

The end-to-end workflow is:

1. Choose the simulation settings in settings.json.
2. Set up the Python environment.
3. (Optional) install Julia and PauliPropagation.jl.
4. Run the benchmarks to produce results.json.
5. Plot the results to produce results.png.

1. Choose the simulation settings

All simulation parameters live in settings.json and are shared by every engine (Python and Julia alike), so a single edit compares them all on an identical problem instance:

Edit the values directly in settings.json before running the benchmarks — e.g. increase nx/ny for a larger system, widen step_min/step_max to simulate more Trotter steps, or relax lower_atol/cutoff to trade accuracy for speed. All engines read this same file, so no code changes are needed to reproduce a benchmark under different conditions.

Qiskit pauli-prop is the one exception: its propagate_through_circuit API has no weight-based cutoff, only a mandatory positive max_terms (which also caps its memory pre-allocation, so it can't be left unbounded). run_model.py sets it to monoprop's own term count at each step, so its per-step term budget tracks cutoff/lower_atol indirectly through monoprop rather than directly.

2. Set up the Python environment

# Navigate to this directory
cd benches/third_party

# Create the venv and install all Python dependencies (including monoprop itself, in editable mode)
uv sync

This installs monoprop, qiskit, ppvm, pauli-prop, and cuquantum-python-cu12. Running the cuPauliProp (GPU) engine requires an NVIDIA GPU with a working CUDA setup; the other three Python engines (monoprop, QuEra ppvm, Qiskit pauli-prop) run on CPU only.

3. (Optional) install Julia and PauliPropagation.jl

Skip this step if you only want to compare the Python engines — run_model.py and plot_results.py work fine without it. Install it if you also want the PauliPropagation.jl comparison:

# Install juliaup and the latest stable Julia release
curl -fsSL https://install.julialang.org | sh

Once Julia has been installed, add some utils:

julia -e 'using Pkg; Pkg.add(["JSON", "ProgressMeter"])'

Finally, install the package PauliPropagation.jl v0.7.3:

julia -e 'using Pkg; Pkg.add(Pkg.PackageSpec(name="PauliPropagation", version="0.7.3"))'

4. Run the benchmarks

# Run the Python engines (monoprop, QuEra ppvm, Qiskit pauli-prop, cuPauliProp) — (re)writes results.json from scratch
uv run python run_model.py

# (Optional) run the Julia engine — merges its results into the existing results.json
julia run_model.jl

run_model.py must be run first: it (re)creates results.json from settings.json. run_model.jl then reads that file and adds the PauliPropagation.jl entries to it, so re-running run_model.py afterwards overwrites the file and drops them again — rerun run_model.jl too if you do.

For every engine, results.json collects, indexed by Trotter step:

• num_terms: the number of Pauli/Majorana terms kept in the evolving operator, for every step.
• runtime: wall-clock time per step, in seconds. The first step is excluded: it carries each engine's warm-up (JIT compilation, first-touch allocation), which is not what the per-step comparison is measuring.
• memory: the memory footprint of the evolving operator, in megabytes, for every step. Where an engine exposes its own accounting this is exact (monoprop's C++ operator-memory accounting, cuPauliProp's cupy device memory pool, PauliPropagation.jl's Base.summarysize of the Pauli sum); QuEra ppvm and Qiskit pauli-prop expose no such accounting, so their footprint is reconstructed by accumulating this process's host-memory growth across each of their own steps.
• expvals: the ZZ expectation value on obs_qubits, for every step.

5. Plot the results

uv run python plot_results.py

This reads results.json and produces a single image, pauli_results.png, combining two log-scale panels (runtime per Trotter step and operator memory footprint per Trotter step) side by side, for each engine present in the file (so they reflect whichever engines you actually ran in step 4).

Results

Runtime and memory footprint per Trotter step for the 2D TFIM problem above, comparing monoprop against cuPauliProp (GPU), PauliPropagation.jl, QuEra ppvm, and Qiskit pauli-prop. monoprop tracks the GPU-accelerated cuPauliProp closely while running on CPU, and outperforms all other CPU-only engines in both time and memory as the operator grows.

Internal benchmarks

The monoprop repository includes a pytest suite measuring the time and peak resident memory (RSS) of monoprop's core operations. The suite is separated from the test suite and can be run with just bench. See below for more detailed instructions.

Running

Each run takes a label and becomes one column in results/REPORT.md, so serial, MPI, and thread variants sit side by side. Extra args go to pytest.

uv sync --group bench                          # once

just bench serial                              # serial run, column "serial"
just bench-smoke                               # quick sanity check (tiny sizes)
just bench serial --num-modes 64 --bench-rounds 10

monoprop_NUM_THREADS=10 just bench serial-t10  # cap the partition count

uv run --group bench python benches/report.py  # rebuild report, no re-run

MPI

Communicator-aware: operations are barrier-wrapped so the timed cost is the makespan across ranks, and memory is the peak of the RSS summed across ranks (shared pages counted per rank, so an upper bound), not the sum of per-rank peaks.

just bench-build-mpi                           # build once (MPI on)
monoprop_NUM_THREADS=2 just bench-mpi r5t2 5 --map-by slot:PE=2 --bind-to core

The MPI build must stay loaded. monoprop builds MPI=OFF by default; a non-MPI extension ignores the communicator, so every rank holds the full operator (N× memory). The recipes use --no-sync so the MPI build survives — rebuild after editing sources. just bench-mpi checks monoprop.has_mpi and aborts if MPI is missing; extra args are forwarded to mpiexec (as root, add --allow-run-as-root).

Benchmarks

Random (bench_random.py; configurable, both pictures) — build_graph, pare, energy, gradient (graph-based), and inplace. CLI options:

Models (bench_models.py; fixed, in-place, slow):

• test_model[hubbard] — 120-qubit Fermi-Hubbard, 29-step Trotter.
• test_model[pauli] — 127-qubit Pauli-basis kicked-Ising, 20 layers.

Override any config field via --<model>-<field> (e.g. --pauli-num-layers 30); just bench --help lists all.

Output

Each run writes results/time-<label>.json (pytest-benchmark) and <label>.json (everything else); report.py merges all labels into results/REPORT.md.