Ai Testing Python Packages

langwatch-scenario

Agentic testing for agentic codebases

prompture

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

giskard

🐢 Open-Source Evaluation & Testing library for LLM Agents

alignmenter

Check if your AI sounds like your brand, stays safe, and behaves consistently. Works with your custom GPTs, hosted APIs, and local models. Get detailed reports in minutes, not days.

evalshift

Open-source LLM migration and regression testing for AI agents. Compare models, detect tool-call regressions, and gate model changes in CI.

seleniumboot-mcp

A Python MCP server that brings Selenium WebDriver automation to Claude and other AI assistants — with built-in code generation for Python (pytest), Java TestNG, and JUnit 5 test scripts.

juried

Acceptance testing for LLM features, with verdicts corrected for the judge's own error rate.

trace2eval-cli

属于 AI 的错题本 · Turn production LLM traces into a regression eval set

langtest

Deliver safe & effective language models

aat-devqa

AI-powered automated E2E testing. Just enter a URL — AI generates and runs test scenarios.

evalcraft

Deterministic $0 pytest assertions for AI agents — catch the agent that quietly stopped calling its tools, and cost blowups, offline in CI.

aiexpect

Assertions for non-deterministic AI text. Drop into pytest, get a Trust Score + HTML report. Free-first: rules, local embeddings, then your own LLM judge.

promptdrift-ci

Git-native CI regression testing and behavioral drift monitoring for LLM prompts.

evalshift-sdk

Open-source capture SDK for AI agents — records model and tool calls to build golden suites for LLM migration and regression testing.

ailens-evals

Evals for AI apps that make text, images, video or audio: is every change better or worse, why, and what it costs. pip install ailens-evals

aitest-kit

一套 AI 辅助的自动化测试工具链

chatbot-tracer

An automated approach for exploring and testing conversational agents using large language models. TRACER discovers chatbot functionalities, generates user profiles, and creates comprehensive test suites for conversational AI systems.

flakelens

Separate real LLM failures from flaky ones. Repeat-run evals with Wilson confidence intervals. Zero deps.

nlptest

Deliver safe & effective language models

haliosai-cli

Open-source CLI and Agent Skill for building and running AI agent evals from your coding agent.

agentrial

Statistical evaluation framework for AI agents

softgnn-advisor

Graph-guided, runtime-proven, LLM-assisted PR testing.

maia-test-framework

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

diffprompt

git diff for prompt engineers - catch behavioural prompt regressions before they ship.