Regression Testing Python Packages

shadow-diff

Behavior contracts for AI agents

pytest-accept

A pytest plugin for automatically updating doctest outputs

reframe-hpc

A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems.

mcp-assert

The deterministic testing standard for MCP servers. Connect over real stdio/SSE/HTTP transport, call tools with real arguments, assert results with 18 assertion types defined in YAML. Any language, any transport, no mocks. Single Go binary.

pytest-mcp-assert

The deterministic testing standard for MCP servers. Connect over real stdio/SSE/HTTP transport, call tools with real arguments, assert results with 18 assertion types defined in YAML. Any language, any transport, no mocks. Single Go binary.

evalview

Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

pytest-ranking

A Pytest plugin for faster fault detection via regression test prioritization

socx-cli

Unified command-line tool for EDA development teams to streamline common tasks and tools, and unify them under a single configurable CLI menu to increase accessibility and transparency of tools and scripts in collaborative development environments.

fhe-attack-replay

Unified attack-replay regression harness for FHE libraries (SEAL, OpenFHE, Lattigo, tfhe-rs).

agentassay

Token-efficient stochastic testing for AI agents. 5-20x cost reduction. 10 framework adapters. Paper: arXiv:2603.02601

touca

Touca SDK for Python

longprobe

Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.

modeldiffx

Behavioral regression testing for LLMs. Capture outputs, diff behavior, detect drift — pytest for model upgrades.

langchain-longprobe

LangChain integration for LongProbe — sub-second RAG retrieval regression testing with chunk-level diffing

conport

Generate continuous testing report

testdatatable

A shared data table store for use by testing applications

webqa-agent

Autonomous web browser agent that audits performance, functionality & UX for engineers and vibe-coding creators. 网站自主评估测试 Agent，支持 GUI/CLI 一键完成性能、功能使用与交互体验的测试评估

touca-fbs

Auto-generated python implementation of Touca FlatBuffers schema

tlscanary

TLS/SSL Test Suite for Mozilla Firefox

promptforge-llmops

Open-source LLMOps framework for prompt versioning, evaluation and regression testing in CI/CD pipelines.

nrtest

Numerical regression testing

tddf

Behaviour regression tests for AI agents — deterministic, local-first, no LLM-as-judge

locomotive

Load and regression performance testing for CI/CD pipelines

falsifyai

Falsification-first reliability testing for AI systems: perturb inputs, preserve replayable evidence, diff reliability across model changes.