Benchmark Python Packages

swebench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

pytest-benchmark

pytest fixture for benchmarking code

mteb

MTEB: Massive Text Embedding Benchmark

pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.

asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

evo

Python package for the evaluation of odometry and SLAM

motmetrics

:bar_chart: Benchmark multiple object trackers (MOT) in Python

google-benchmark

A microbenchmark support library

pyperformance

Python Performance Benchmark Suite

picows

Ultra-fast websocket client and server for asyncio

membrowse

Track and analyze binary size and memory footprint in embedded firmware

beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

medmnist

[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification

mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.

optunahub

Python library to use and implement packages in OptunaHub

pytest-django-queries

Generate performance reports from your django database performance tests.

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

bigcodebench

[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI

agentdojo

A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

holobench

A package for benchmarking the characteristics of arbitrary functions

chessboard

:game_die: CLI to solve combinatoric chess puzzles.

apebench

[Neurips 2024] A benchmark suite for autoregressive neural emulation of PDEs. (≥46 PDEs in 1D, 2D, 3D; Differentiable Physics; Unrolled Training; Rollout Metrics)

fjsplib

Python package to read and write instances for the flexible job shop problem.

optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.