Ai Evaluation Python Packages

uqlm

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

eval-ai-library

Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.

agent-action-guard

🛡️ Safe AI Agents through Action Classifier

gauntlet-cli

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

nasde-toolkit

CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

grandjury

Python SDK for HumanJudge — real human evaluations of AI models. 25,000+ blind reviews by 200+ verified reviewers across 58 models and 44 benchmarks. Free.

gaico

A Python library providing evaluation metrics to compare generated texts from LLMs, often against reference texts. Features streamlined workflows for model comparison and visualization.

evalstats

Statistically sane analysis methods for comparing AI model and prompt performance.

workflowbench

Lightweight benchmark harness for AI-driven business workflows

ai-stability

Measure LLM output consistency from the command line.

promptstats

Statistical analysis methods for comparing prompt and model performance in LLM evaluations.

iaa-kit

Modern inter-annotator agreement metrics with bootstrap intervals, ordinal support, and missing-data handling.

eval-run-manifest

Portable manifest envelope for eval run provenance, artifacts, and reproducibility.

judge-bench

Bias probes and reproducible diagnostics for LLM-as-judge evaluation workflows.

judge-card

A disclosure format for judge prompts, calibration results, known bias, and recommended use envelopes.

eval-adapter

Adapters between rubric-spec and common evaluation framework inputs.

contamination-audit

Local contamination checks for eval data overlap, hashes, and n-gram leakage.

synthetic-disagreement

Synthetic reviewer disagreement generators for testing IAA and adjudication workflows.

aisert

Assert-style validation library for AI outputs - ensure your LLMs behave exactly as expected.

chatbot-auditor

Quality auditor for AI chatbots. Analyzes your conversation logs to show where the bot is underperforming.

rail-score

DEPRECATED — use rail-score-sdk instead. This package redirects to rail-score-sdk.

factly-eval

CLI tool to evaluate LLM factuality on MMLU benchmark.