Llm Testing Python Packages

evalshift

Open-source LLM migration and regression testing for AI agents. Compare models, detect tool-call regressions, and gate model changes in CI.

juried

Acceptance testing for LLM features, with verdicts corrected for the judge's own error rate.

langtest

Deliver safe & effective language models

evalcraft

Deterministic $0 pytest assertions for AI agents — catch the agent that quietly stopped calling its tools, and cost blowups, offline in CI.

aqven-llm

Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents

aqven

Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents

evalshift-sdk

Open-source capture SDK for AI agents — records model and tool calls to build golden suites for LLM migration and regression testing.

aquamind

LLM 应用在并发阶梯负载下的质量变化判定工具（建设中） —— 回答 "这个下降是不是真的"。

agentneo

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

stepfork

Open-source Python toolkit to record and replay AI agent executions, detect regressions, and turn failures into reproducible pytest tests.

agent-trajectory-diff

Compare AI agent execution paths (trajectories) side-by-side. Detect trajectory drift, redundant tool loops, and resource regressions in CI/CD.

proofagent-harness

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

pytest-mockllm

Deterministic pytest mocks for OpenAI, Anthropic, Gemini, and LangChain—test LLM apps without live API calls.

llamator

Red Teaming python-framework for testing chatbots and GenAI systems.

narrative-contracts

MateProbe by Mate4B — mutation testing for your validators. Audit missed faults, wrong-reason rejections and valid controls with pytest.

pytest-narrative-contracts

MateProbe by Mate4B — mutation testing for your validators. Audit missed faults, wrong-reason rejections and valid controls with pytest.

ai-safety-tester

LLM Security Testing Framework | CVE-style severity scoring, prompt injection detection, bias analysis | Strict mypy typing | Fast CI/CD with pytest markers | PyPI: ai-safety-tester

gauntlet-cli

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

agentassay

Token-efficient stochastic testing for AI agents. 5-20x cost reduction. 10 framework adapters. Paper: arXiv:2603.02601

fakellm

Mock OpenAI and Anthropic APIs locally for fast, deterministic tests.

nlptest

Deliver safe & effective language models

api-test-ninja

API Ninja simplifies API testing by allowing users to define test flows in plain English.

agent-eval-runner

5-case free starter for the AI Agent QA Eval Pack. Vendor-agnostic YAML eval cases for tool-using LLM agents (LangChain / OpenAI / Anthropic / custom). CC BY 4.0.

mateprobe

MateProbe by Mate4B — mutation testing for your validators. Audit missed faults, wrong-reason rejections and valid controls with pytest.