Regression Testing Python Packages

pytest-accept

A pytest plugin for automatically updating doctest outputs

reframe-hpc

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.

wup

WUP (What's Up) - Intelligent file watcher for regression testing in large projects

shadow-diff

Behavior contracts for AI agents

mcp-assert

The testing standard for MCP servers. Lint (14 rules), test, and fuzz over real stdio/SSE/HTTP transport. 18 assertion types in YAML. Found 4,794 schema issues across 55 servers. Any language, no mocks, single binary.

evalview

Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

falsifyai

Portable, content-addressed reliability evidence for LLM systems. Capture how a model behaves under perturbation; preserve, verify, and diff the evidence across model changes.

robotrace-dev

Python SDK for RoboTrace: observability and evals for AI-powered robots. Log episodes (synchronized video + sensors + actions) from rosbag2 or Hugging Face LeRobot datasets, version them, and re-roll new policy versions against historical observations.

dprovenancekit

Reasoning observability and regression testing for AI systems — a Python port of DProvenanceKit.

agentdelta

Diff and regression-detect LLM agent execution traces

pytest-mcp-assert

The testing standard for MCP servers. Lint (14 rules), test, and fuzz over real stdio/SSE/HTTP transport. 18 assertion types in YAML. Found 4,794 schema issues across 55 servers. Any language, no mocks, single binary.

pytest-ranking

A Pytest plugin for faster test failure detection

tracelens

Friendly evaluation and regression-testing framework for AI agents: inspectable traces, graded outcomes, baseline comparisons, and CI-ready reliability signals.

ciagent

Pytest-native regression testing for AI agents. Catch tool-call drift and cost spikes before production.

socx-cli

Unified command-line tool for EDA development teams to streamline common tasks and tools, and unify them under a single configurable CLI menu to increase accessibility and transparency of tools and scripts in collaborative development environments.

modelpin

Dependabot for AI models — replay your app's scenarios across a model migration and catch real behavioral regressions before they ship. Low false-positive by design. CLI + GitHub Action. Open-core, Apache-2.0.

touca

Continuous Regression Testing for Engineering Teams

modeldiffx

Model behavioral diffing - compare LLM outputs across versions, detect regressions.

webqa-agent

Autonomous web browser agent that audits performance, functionality & UX for engineers and vibe-coding creators. 网站自主评估测试 Agent，支持 GUI/CLI 一键完成性能、功能使用与交互体验的测试评估

agentassay

Formal regression testing for non-deterministic AI agent workflows

testdatatable

A shared data table store for use by testing applications

recalllab

pytest for agent memory: turn memory expectations into pytest contracts that run in CI.

redline-ai

Catch prompt regressions before they ship, using logs you already have.

touca-fbs

Continuous Regression Testing for Engineering Teams