Ai Evaluation Python Packages

calibrate-sdk

Python SDK for Calibrate's API

eval-ai-library

Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.

clinical-ai-reliability

Open-source reliability framework for medical and health AI systems.

bohrin

Check the checker: an open standard for testing the graders of RL environments, with proof before every accusation.

cje-eval

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

proofbundle

Portable evidence for AI work, verifiable offline. Integrity, not truth

multivon-eval

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

uqlm

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

agentverity

Qualify repeated categorical AI-agent evidence before it becomes a regression reference

evalstats

Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

environment-harness

Run agents in persistent shared environments with private observations, checkpoints and auditable evidence.

ifixai

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

agent-action-guard

🛡️ Safe AI Agents through Action Classifier

evalt

Budget-bounded LLM routing that finds the cheapest model and prompt meeting your accuracy target.

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

touchstone-dqi

An open-source AI evaluation tool that makes results easy to trust and verify. Every result includes its confidence range and sample size, and everything is saved in an evidence bundle that anyone can check offline.

dspy-security-bench

Open, privacy-bounded assurance for AI agents: containment provenance, identity passports, authorization twins, OTel evidence, CI gates, and OSCAL.

falsiflow

Stop unverifiable AI eval, product metric, and R&D claims from passing CI.

evalforge-ci

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

pytest-judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

journeyman-bench

A process-quality benchmark for LLM agents: it grades how an agent worked, not whether it passed — and every LLM-as-a-judge sits a calibrated exam before it may score.

gauntlet-evals

pip install gauntlet-evals · v0.1.0. Merge-blocking evaluation gates for generative AI features: YAML suites run against any HTTP endpoint or Python callable, fail the build on a miss, and emit both a diffable JSON pack and a reviewer document cross-referenced to California's published GenAI risk framework. Aligned to, never approved by.

nobulex

Reproducible research on verification boundaries, plus the earlier Nobulex receipt prototype. Current financial decision-integrity gateway: github.com/arian-gogani/nobulex-registry.