Benchmark Python Packages

swebench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

pytest-benchmark

pytest fixture for benchmarking code

mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.

fjsplib

Python package to read and write instances for the flexible job shop problem.

asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

evo

Python package for the evaluation of odometry and SLAM

motmetrics

:bar_chart: Benchmark multiple object trackers (MOT) in Python

membrowse

Track and analyze binary size and memory footprint in embedded firmware

agentdojo

A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

google-benchmark

A microbenchmark support library

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

chessboard

:game_die: CLI to solve combinatoric chess puzzles.

beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

picows

Ultra-fast websocket client and server for asyncio

mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.

optunahub

Python library to use and implement packages in OptunaHub

medmnist

[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification

pytest-django-queries

Generate performance reports from your django database performance tests.

holobench

A package for benchmarking the characteristics of arbitrary functions

pyperformance

Python Performance Benchmark Suite

apebench

[Neurips 2024] A benchmark suite for autoregressive neural emulation of PDEs. (≥46 PDEs in 1D, 2D, 3D; Differentiable Physics; Unrolled Training; Rollout Metrics)

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

causal-worlds

Generate fictional-but-coherent causal operations worlds (executable sim + time-series + ground-truth causal answer-key) from a natural-language description — for benchmarking causal-discovery and control agents.