Llm Evals Python Packages

whileai

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

zeroproof

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

mcp-tool-evals

Behavioral evaluation and description linting for MCP servers

calibrate-sdk

Python SDK for Calibrate's API

trace2eval-cli

属于 AI 的错题本 · Turn production LLM traces into a regression eval set

agentdog

AgentDog helps developers inspect, test, score, and monitor AI agent runs locally.

aqven-llm

Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents

aqven

Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents

tidy-ai

Clean up Gmail and YouTube with AI, safely: Jev judges each email or channel, plain code sets the limits, you approve. pipx install tidy-ai

judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

pytest-judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

eval-conductor

Claude Code plugin: fail-closed eval release gates + an evidence-gated 27-step build ledger. NaN and stale baselines BLOCK; placeholder evidence is refused. /eval-init installs it into any repo.

lcb-gate

Statistical pass/fail gating for flaky tests and LLM evals: Wilson lower-bound verdicts, paired A/B, pytest plugin, stdlib only

journey-evals

Drive a real browser or a real LangGraph agent through one declared user journey, and report what actually happened. A page saying 'success' is never accepted as proof.

laya-evals

LLM-as-judge and calibration auditing for System 1 decision models: rubric scoring in CI plus reliability diagrams, ECE and coverage/accuracy thresholds by option count

ai-stability

Measure LLM output consistency from the command line.

apte

Async-first Python testing framework with explicit dependency injection and native LLM evals. Tests and evals run as coroutines on one event loop.

mcp-bigquery-evals

Read-only BigQuery MCP server with mandatory dry-run cost caps, agent-friendly structured errors, and a Spider/BIRD-style NL-to-SQL eval harness.

evalops

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

dormant-behavior-audit

Public benchmark and reference bundle for dormant behavior audit

zeroproof-simulations

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

tripwire-oracle

Correctness oracle for AI code-optimization loops: catches reward hacks with canonical, metamorphic, withheld-input, and timing checks.