Benchmarking Python Packages

pytest-benchmark

pytest fixture for benchmarking code

locust

Write scalable load tests in plain Python 🚗💨

pytest-codspeed

A pytest plugin to create benchmarks

pyperf

Toolkit to run Python benchmarks

asv-runner

Core pure python benchmarking for asv

openml

OpenML's Python API for a World of Data and More 💫

opendsstar

OpenDsStar is an open-source implementation of the DS-Star agent that replaces file-based workflows with a flexible, tool-centric architecture. It supports incremental execution, reuses intermediate results, and makes complex multi-step agents more modular, efficient, and extensible.

openjij

OpenJij : Framework for the Ising model and QUBO.

pytest-django-queries

Generate performance reports from your django database performance tests.

fev

Forecast evaluation library

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

holobench

A package for benchmarking the characteristics of arbitrary functions

reframe-hpc

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.

salesforce-merlion

Merlion: A Machine Learning Framework for Time Series Intelligence

arthur-observability-sdk

Make AI work for Everyone - Monitoring and governing for your AI/ML

benchexec

BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

isat-tuner

Inference Stack Auto-Tuner — one command to find the fastest ONNX inference config for any model on any GPU. Bayesian optimization, Pareto analysis, thermal-aware benchmarking, multi-provider (ROCm/CUDA/TensorRT/OpenVINO).

autorag

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

cmind

Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks

easy-tpp

EasyTPP: Towards Open Benchmarking Temporal Point Processes

pyperform

An easy and convienent way to performance test python code.

perun

Measure the energy used by your MPI+Python applications.

immuneml

immuneML is a platform for machine learning analysis of adaptive immune receptor repertoire data.

pythae

Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)