Benchmark Framework Python Packages

pytest-django-queries

Generate performance reports from your django database performance tests.

optiprofiler

Benchmarking optimization solvers.

holobench

A package for benchmarking the characteristics of arbitrary functions

cirq-superstaq

https://superstaq.readthedocs.io

checks-superstaq

https://superstaq.readthedocs.io

general-superstaq

https://superstaq.readthedocs.io

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

qiskit-superstaq

https://superstaq.readthedocs.io

supermarq

https://superstaq.readthedocs.io

lakebench

A multi-modal Python library for benchmarking lakehouse engines and ELT scenarios, supporting both industry-standard and novel benchmarks.

langtest

Deliver safe & effective language models

timeeval

Evaluation Tool for Anomaly Detection Algorithms on Time Series

running-ng

Running next generation

bayesmark

Benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks

banana-hep

Benchmarking AgaiNst Apfel aNd Anything

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

apolien

AI Safety Evaluation Library

benched

Easy benchmarking

mlip-arena

🌟 [NeurIPS '25 Spotlight] Fair and transparent benchmark of machine learning interatomic potentials (MLIPs), beyond basic error metrics https://openreview.net/forum?id=SAT0KPA5UO

decent-bench

A benchmarking framework for decentralized optimization

nlptest

Deliver safe & effective language models

agentcompass

[EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.

enfobench

Energy Forecast Benchmark Toolkit is a Python project that aims to provide common tools to benchmark forecast models.

rllte-core

Long-Term Evolution Project of Reinforcement Learning