Model Evaluation Python Packages

canary-ml

Drift and anomaly detection for production ML models

falsify

A single-file Python CLI that pre-registers AI/ML accuracy claims with SHA-256. Lock the threshold before the data, or it didn't happen.

ml4t-diagnostic

Signal diagnostics, statistical validation, and backtest evaluation for quantitative trading workflows.

ecp-runtime

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

ecp-sdk

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

pyaerocom

Python tools for climate and air quality model evaluation

falsify-inspect

PRML pre-registration adapter for Inspect AI eval logs. MIT.

debiai-gui

DebiAI easy start module, the standalone version of DebiAI

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

mlflow-falsify

A single-file Python CLI that pre-registers AI/ML accuracy claims with SHA-256. Lock the threshold before the data, or it didn't happen.

arxiv-embedding-benchmark

Published PyPI package for ArXiv embedding benchmarks, retrieval evaluation, and scientific RAG experiments.

starwhale

An MLOps Platform for Model Evaluation

evalcards

Librería Python para generar reportes de evaluación (clasificación, regresión, forecasting) con métricas y gráficos listos en Markdown, JSON y pronto HTML.

oncothresh

Clinical threshold evaluation for oncology AI models

trustlens

Audit ML models beyond accuracy — calibration, fairness, latent health, and deployment verdicts.

starwhale-bootstrap

MLOps Platform

modeldiffx

Model behavioral diffing - compare LLM outputs across versions, detect regressions.

grandjury

GrandJury SDK — submit LLM traces for human evaluation + analytics client

hermia

Vendor-agnostic security evals for LLM inference stacks. Detects behavioral divergence across models, backends, and hardware.

model-failure-lab

Toolkit for systematically probing, classifying, and debugging failure modes in LLM and RAG systems — reasoning errors, hallucination, and API-level behaviour.

slicemap-cli

Find the data slices where a new model regressed against an old one.

insurance-cv

Temporal and distributional cross-validation, and feature screening, for insurance pricing models

data-science-snippets

A modular set of data science utilities for EDA, cleaning, and more.

easymlselector

A model selection process for Machine Learning tasks on subset of training sample