Model Evaluation Python Packages

hotcoco

Perception evaluation toolkit in Rust with Python bindings. A pycocotools drop-in for COCO, LVIS, and Open Images detection metrics, up to 36× faster, plus TIDE error analysis, confusion matrices, calibration, model comparison, and a dataset browser.

calibrax

JAX benchmarking, profiling and evaluation metrics for Flax NNX: a registry of pure-function metrics (regression, classification, calibration, uncertainty, forecasting, generative, image, text, audio, graph, fairness), XLA FLOP counting, roofline analysis, GPU and energy monitoring, regression detection, publication exports, W&B and MLflow.

ml4t-diagnostic

Signal diagnostics, statistical validation, and backtest evaluation for quantitative trading workflows.

flac-detective

Catches music sold as lossless that was secretly compressed. It accuses only when independent lines of evidence agree, because a false accusation costs more than a miss.

evalarc

Review model and agent regressions behind better scores. Compare every changed check, inspect original model outputs and tool traces, and reproduce the decision offline.

evalshift

Open-source LLM migration and regression testing for AI agents. Compare models, detect tool-call regressions, and gate model changes in CI.

claude-drift

Replay your own Claude Code sessions against a new model and see what actually changed

pyattacker

Resumable async task orchestration for Python — pipelines, artifact persistence, per-endpoint pools. One failure costs a task, not the run.

poseaudit

Audit how far off the angles, tilts and lengths a pose model measures are

hermia

Vendor-agnostic security evals for LLM inference stacks. Detects behavioral divergence across models, backends, and hardware.

yolozu

ToppyMicroServices commercial product, provided free of charge, for validating and fairly evaluating existing vision predictions through a stable predictions interface contract.

decishift

Open-source ML behavioral regression testing for comparing decision-system versions, identifying changed actions, attributing shifts, and gating releases with Decision Contracts.

evalshift-sdk

Open-source capture SDK for AI agents — records model and tool calls to build golden suites for LLM migration and regression testing.

falsify

Fix the pass bar before the test runs. PRML is an open spec and CLI that hashes and timestamps evaluation criteria (SHA-256, RFC 3161) so anyone can check them later. Python, JS, Go, Rust.

touchstone-dqi

An open-source AI evaluation tool that makes results easy to trust and verify. Every result includes its confidence range and sample size, and everything is saved in an evidence bundle that anyone can check offline.

modelscout-llm

ModelScout checks your hardware and gives you a shortlist before you download the weights. You can scan your own machine or supply a different GPU and memory size to see what might fit.

senbonzakura

The world's top open-weight AI model workshop. Measure, uncensor and audit: refusal removal for transformer language models, with the instruments to measure what the edit cost.

actseal

Test frozen model decision policies with risk/coverage bounds, provider-fault checks and offline replay.

pyaerocom

Python tools for climate and air quality model evaluation

debiai-gui

Bias detection and contextual evaluation tool for your AI projects

starwhale-bootstrap

an MLOps/LLMOps platform

mlx-quant-fidelity

Measure quantization quality loss on Apple Silicon MLX — KL divergence, top-token flip rate and perplexity delta for KV-cache and weight quantization

starwhale

an MLOps/LLMOps platform

senbonzakura-check

The world's top open-weight AI model workshop. Measure, uncensor and audit: refusal removal for transformer language models, with the instruments to measure what the edit cost.