Ai Testing Python Packages

langwatch-scenario

Agentic testing for agentic codebases

giskard

🐢 Open-Source Evaluation & Testing library for LLM Agents

prompture

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

aat-devqa

AI-powered automated E2E testing. Just enter a URL — AI generates and runs test scenarios.

langtest

Deliver safe & effective language models

nlptest

Deliver safe & effective language models

chatbot-tracer

An automated approach for exploring and testing conversational agents using large language models. TRACER discovers chatbot functionalities, generates user profiles, and creates comprehensive test suites for conversational AI systems.

llm-behave

Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.

intentguard

A Python library for verifying code properties using natural language assertions.

longprobe

Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.

agentrial

Statistical evaluation framework for AI agents - pytest for agent trajectories

alignmenter

Check if your AI sounds like your brand, stays safe, and behaves consistently. Works with your custom GPTs, hosted APIs, and local models. Get detailed reports in minutes, not days.

py-toolguard

Cloudflare for AI Agents. 7-layer security interceptor and observability dashboard.

aetherlab

Open-source tools, SDKs, and resources for AetherLab AI quality control platform

tenro

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

maia-test-framework

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

llm-sentry

Unified AI Reliability Platform. One install, 12 diagnostic engines. Zero-dependency LLM pipeline monitoring.

ccheck

A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.

sharingan-autotest

Autonomous testing agent for Claude Code. Discovers, tests, diagnoses, and fixes your web app.

pyllmtest

🚀 Comprehensive testing framework for LLM applications with semantic assertions, multi-provider support, RAG testing, and prompt optimization. Test AI the right way!

llmtest-framework

pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions

llmtestr

A new package that helps developers integration-test AI and LLM applications by validating structured outputs. It takes a user's test scenario or prompt as input, sends it to an LLM, and uses pattern

forge-nc

Local-first AI coding assistant with 9-layer security, adversarial model testing, and cryptographic audit trails. Break your AI before it breaks your code.

housemonkey

Chaos testing for AI apps. 18 extreme personas attack your AI to find edge cases before users do. OWASP LLM Top 10 coverage.