Benchmark Python Packages

swebench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

20.4M 6K 1K

pytest-benchmark

pytest fixture for benchmarking code

10.3M 1K 139

mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

698K 3K 719

pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.

415K 77 10

trajectory-sdk

Practical recipes for adapting benchmarks, training models, and measuring reward with the Trajectory SDK.

341K 0 0

asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

297K 1K 212

evo

Python package for the evaluation of odometry and SLAM

173K 4K 801

picows

Ultra-fast websocket client and server for asyncio

86K 293 21

motmetrics

:bar_chart: Benchmark multiple object trackers (MOT) in Python

83K 1K 261

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

72K 2K 209

beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.

60K 2K 254

astral-dev-toolchain-hyperfine

A command-line benchmarking tool

55K 29K 522

pyperformance

Python Performance Benchmark Suite

53K 1K 204

pytest-django-queries

Generate performance reports from your django database performance tests.

52K 83 2

membrowse

Track and analyze binary size and memory footprint in embedded firmware

47K 45 2

optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

31K 340 64

agentdojo

A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents.

28K 906 242

optunahub

Python library to use and implement packages in OptunaHub

28K 63 17

optiprofiler

Benchmarking optimization solvers.

28K 16 5

apebench

[Neurips 2024] A benchmark suite for autoregressive neural emulation of PDEs. (≥46 PDEs in 1D, 2D, 3D; Differentiable Physics; Unrolled Training; Rollout Metrics)

27K 109 2