Model Evaluation Python Packages

debiai-gui

DebiAI easy start module, the standalone version of DebiAI

gauntlet-cli

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

falsify

A single-file Python CLI that pre-registers AI/ML accuracy claims with SHA-256. Lock the threshold before the data, or it didn't happen.

pyaerocom

pyaerocom model evaluation software

starwhale-bootstrap

an MLOps/LLMOps platform

trustlens

Open-source Python library for evaluating ML model reliability beyond accuracy — with calibration, failure, and fairness diagnostics for informed deployment decisions.

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

starwhale

an MLOps/LLMOps platform

ecp-sdk

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

ecp-runtime

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

grandjury

Python SDK for HumanJudge — real human evaluations of AI models. 25,000+ blind reviews by 200+ verified reviewers across 58 models and 44 benchmarks. Free.

mlflow-falsify

A single-file Python CLI that pre-registers AI/ML accuracy claims with SHA-256. Lock the threshold before the data, or it didn't happen.

model-failure-lab

Toolkit for discovering, classifying, and debugging failure modes in LLM and RAG systems.

modeldiffx

Behavioral regression testing for LLMs. Capture outputs, diff behavior, detect drift — pytest for model upgrades.

easymlselector

A model selection process for Machine Learning tasks on subset of training sample

insurance-cv

Temporal cross-validation for insurance pricing - respects policy time structure, CatBoost, Polars

llm-sentry

Unified AI Reliability Platform. One install, 12 diagnostic engines. Zero-dependency LLM pipeline monitoring.

titus2

Titus 2 : Portable Format for Analytics (PFA) implementation for Python 3.4+

data-science-snippets

🧰 Essential EDA and Data Cleaning Helpers for Any DataFrame This collection of functions is designed to accelerate exploratory data analysis (EDA), quickly surface data quality issues, and offer high-level insights into the structure and content of your dataset.

evalcards

Librería Python para generar reportes de evaluación (clasificación, regresión, forecasting) con métricas y gráficos listos en Markdown, JSON y pronto HTML.

metriculous

Measure and visualize machine learning model performance without the usual boilerplate.

skrobot

skrobot is a Python module for designing, running and tracking Machine Learning experiments / tasks. It is built on top of scikit-learn framework.

scorecardbundle

A High-level Scorecard Modeling API | 评分卡建模尽在于此

machlearn

A Simple Yet Powerful Machine Learning Python Library