Benchmarking Python Packages

pytest-benchmark

pytest fixture for benchmarking code

locust

Write scalable load tests in plain Python 🚗💨

pytest-codspeed

A pytest plugin to create benchmarks

pyperf

Toolkit to run Python benchmarks

asv-runner

Core pure python benchmarking for asv

astral-dev-toolchain-cargo-codspeed

Crates to benchmark your Rust code

openml

OpenML's Python API for a World of Data and More 💫

fev

Forecast evaluation library

pytest-django-queries

Generate performance reports from your django database performance tests.

openjij

OpenJij : Framework for the Ising model and QUBO.

data-foundry

Repository for all things around tabular dataset curation for benchmarking!

calibrax

JAX benchmarking, profiling and evaluation metrics for Flax NNX: a registry of pure-function metrics (regression, classification, calibration, uncertainty, forecasting, generative, image, text, audio, graph, fairness), XLA FLOP counting, roofline analysis, GPU and energy monitoring, regression detection, publication exports, W&B and MLflow.

chimeraforge

PyPI capacity-planning CLI for LLM deployment. pip install chimeraforge.

reframe-hpc

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.

holobench

A package for benchmarking the characteristics of arbitrary functions

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

code-mower

AI peer-programmer and reviewer orchestration, calibration, and reporting.

salesforce-merlion

Merlion: A Machine Learning Framework for Time Series Intelligence

mqt-bench

MQT Bench - A Tool for Benchmarking Quantum Software Tools

scitex-benchmark

Performance benchmarking, runtime monitoring, and profiling helpers

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

bexhoma

This python tool helps managing DBMS benchmarking experiments in a Kubernetes-based HPC cluster environment. It enables users to configure hardware / software setups for easily repeating tests over varying configurations.

opendsstar

OpenDsStar is an open-source implementation of the DS-Star agent that replaces file-based workflows with a flexible, tool-centric architecture. It supports incremental execution, reuses intermediate results, and makes complex multi-step agents more modular, efficient, and extensible.

denet

a streaming process monitor