Agent Testing Python Packages

langwatch-scenario

Agentic testing for agentic codebases

mcp-tool-evals

Behavioral evaluation and description linting for MCP servers

flowproof

Agents are starting to run real business processes. flowproof tests them like anything else: record one real run, then replay it on every commit with zero LLM calls, asserting which tools were called, with which arguments, in which order, and which were not.

coder-eval

Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.

skillxp

Skill invocation runner: install an Agent Skill in a fresh fixture, invoke an agent harness headlessly, and report what actually reached the model, with transcript evidence.

fathom-read

Catch the step where an AI agent contradicts its own earlier decision. CLI and adapters for LangGraph, CrewAI, Letta, OpenInference, DBOS and multi-agent logs, plus an expiry read for long-running agents.

evalcraft

Deterministic $0 pytest assertions for AI agents — catch the agent that quietly stopped calling its tools, and cost blowups, offline in CI.

knowlytix-core

Apache-2.0 source for the three-package Knowlytix GMSH agentic-AI testing stack: knowlytix-doc, knowlytix-bench, knowlytix-harness. Release-time mirror of knowlytix/GMS.

a2a-proof

Black-box contract tests for A2A agents

stepfork

Open-source Python toolkit to record and replay AI agent executions, detect regressions, and turn failures into reproducible pytest tests.

proofagent-harness

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

agent-chronicle

Record-and-replay for agent decision graphs: reproduce a prod agent failure as a committed regression test, and re-run your fix without live LLM calls.

agent-regress-cli

Statistical regression testing for LLM agents: p-value, effect size, and 95% CI on real behavior shifts vs. noise.

agentbreak

infrastructure chaos to test the resilience of ai agents

agentos-platform

Deterministic record, replay, and git-bisect for AI agent runs. Reproduce a broken run byte-for-byte offline, then binary search to the commit that broke it.

swarm-test

Chaos engineering & reliability testing for multi-agent AI systems

agentassay

Token-efficient stochastic testing for AI agents. 5-20x cost reduction. 10 framework adapters. Paper: arXiv:2603.02601

agentbarrier

Deterministic approval, rejection, cancellation, timeout, replay, and side-effect safety tests for AI agents—no API keys required.

knowlytix-harness

Apache-2.0 source for the three-package Knowlytix GMSH agentic-AI testing stack: knowlytix-doc, knowlytix-bench, knowlytix-harness. Release-time mirror of knowlytix/GMS.

knowlytix-benchmark

Apache-2.0 source for the three-package Knowlytix GMSH agentic-AI testing stack: knowlytix-doc, knowlytix-bench, knowlytix-harness. Release-time mirror of knowlytix/GMS.

knowlytix-knowledge

Apache-2.0 source for the three-package Knowlytix GMSH agentic-AI testing stack: knowlytix-doc, knowlytix-bench, knowlytix-harness. Release-time mirror of knowlytix/GMS.

agentrunproof

Reproduce OpenAI Agents SDK runtime bugs without API keys — real Runner, scripted models, verifiable certificates.

right-rudder

Catch the step where an AI agent contradicts its own earlier decision. CLI and adapters for LangGraph, CrewAI, Letta, OpenInference, DBOS and multi-agent logs, plus an expiry read for long-running agents.

dryfire

Git-native regression testing for LLM agent tool loops. Assert on the trajectory — the ordered tool calls — not the final prose. Local-first, deterministic, no API key in CI; exit codes are the API.