Agent Evals Python Packages

whileai

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

zeroproof

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

calibrate-sdk

Python SDK for Calibrate's API

haenv

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

clinescope

Cline is an AI coding assistant that keeps a log of what it did. Clinescope reads that log and tells you: whether it used the tools you expected, whether its edit instructions were in the form Cline needs, whether it retyped whole blocks instead of making small changes, and whether it retried the edits that failed.

agentloss

Measure the real-world error rate and dollar cost of an AI agent's decisions. OpenTelemetry-native.

evalctl

Local-first eval harness for agents that scores workspace effects, command logs, and artifacts, not just prompt text.

looplet

Test-driven harness engineering for Python agents: own the loop, capture failures, and gate every change.

evalops

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

nodetracer

The node-level tracing library for agentic software.

zeroproof-simulations

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

repoagentbench

SWE-bench for your codebase — mine your merged PRs into local, contamination-free coding-agent benchmarks. Adapters: claude-code, aider (Opus 4.7 / GPT-5.5 / Sonnet 4.6 / Gemini 3.1 Pro).