Llm Testing Python Packages

proofagent-harness

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

langtest

Deliver safe & effective language models

nlptest

Deliver safe & effective language models

evalcraft

Generate deterministic pytest tests for your AI agents from one real run, then replay them in CI for $0. Fast, flake-free agent testing.

fakellm

Mock OpenAI and Anthropic APIs locally for fast, deterministic tests.

agentneo

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

llamator

Framework for testing vulnerabilities of GenAI systems.

temper-skills

Test suites + deterministic Python for your agent skills' decision logic. Adversarial persona reviewers write the tests; the generated code must keep passing them — zero LLM calls at inference. Try: uvx temper-skills audit <skill.md>

ai-safety-tester

LLM Security Testing Framework | CVE-style severity scoring, prompt injection detection, bias analysis | Strict mypy typing | Fast CI/CD with pytest markers | PyPI: ai-safety-tester

llm-behave

Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.

agent-eval-runner

5-case free starter for the AI Agent QA Eval Pack. Vendor-agnostic YAML eval cases for tool-using LLM agents (LangChain / OpenAI / Anthropic / custom). CC BY 4.0.

agentassay

Formal regression testing for non-deterministic AI agent workflows

api-test-ninja

API Ninja simplifies API testing by allowing users to define test flows in plain English.

infer-check

Correctness and reliability testing for LLM inference engines

recalllab

pytest for agent memory: turn memory expectations into pytest contracts that run in CI.

ragaliq

LLM & RAG evaluation testing framework — hallucination detection, faithfulness metrics, answer relevance scoring, and retrieval pipeline testing with pytest integration

tinyqabenchmarkpp

A tiny synthetic QA LLM benchmark dataset generator using LiteLLM.

replayd

Turn failed AI agent runs into replayable regression tests

ccheck

A human-friendly framework for testing and evaluating LLMs, RAGs, and chatbots.

llm-sentry

Unified AI reliability platform. One install, 12 diagnostic engines. Continuous monitoring, fault diagnosis, and compliance for LLM pipelines.

mocktopus

🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing

pyllmtest

🚀 Comprehensive testing framework for LLM applications with semantic assertions, multi-provider support, RAG testing, and prompt optimization. Test AI the right way!

agentcloudkelp

YAML-first stress testing for AI agents. Inject faults, catch behavioral drift, enforce cost budgets.