Benchmark Framework Python Packages

pytest-django-queries

Generate performance reports from your django database performance tests.

holobench

A package for benchmarking the characteristics of arbitrary functions

general-superstaq

https://superstaq.readthedocs.io

cirq-superstaq

https://superstaq.readthedocs.io

checks-superstaq

https://superstaq.readthedocs.io

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

qiskit-superstaq

https://superstaq.readthedocs.io

supermarq

https://superstaq.readthedocs.io

langtest

Deliver safe & effective language models

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

nlptest

Deliver safe & effective language models

optiprofiler

Benchmarking optimization solvers.

timeeval

Evaluation Tool for Anomaly Detection Algorithms on Time Series

lakebench

A multi-modal Python library for benchmarking lakehouse engines and ELT scenarios, supporting both industry-standard and novel benchmarks.

banana-hep

Benchmarking AgaiNst Apfel aNd Anything

krisis

A clinical evaluation framework for large language models.

rllte-core

Long-Term Evolution Project of Reinforcement Learning

tne-sdk

Python SDK and terminal client for The Null Epoch - an MMO where every player is an AI agent.

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks

apolien

AI Safety Evaluation Library

enfobench

Energy forecast benchmarking toolkit.

running-ng

Running: Next Generation

pymlbenchmark

Benchmarks about Machine Learning in Python.

mlip-arena

🌟 [NeurIPS '25 Spotlight] Fair and transparent benchmark of machine learning interatomic potentials (MLIPs), beyond basic error metrics https://openreview.net/forum?id=SAT0KPA5UO