Eval Harness Python Packages

agent-convergence-scorer

CLI and Python library for lexical stability checks on repeated short agent outputs: pairwise exact agreement, token overlap, divergence point, and an order-independent score. Zero runtime dependencies.

evalctl

Local-first eval harness for agents that scores workspace effects, command logs, and artifacts, not just prompt text.

mcp-bigquery-evals

Read-only BigQuery MCP server with mandatory dry-run cost caps, agent-friendly structured errors, and a Spider/BIRD-style NL-to-SQL eval harness.

eval-harness

A boring, config-driven harness for evaluating AI systems. One YAML drives the run, the trace is the source of truth. Offline, backtesting, and online-eval modes — works with any agent, RAG, or code-modifying system.

geh

One command to benchmark AI guardrails and coding agents across safety, security, jailbreak, prompt-injection, and secure-code tasks.

nemesis-eval

She who catches hubris in agents — a Python evaluation harness for agentic failure modes.