Prompt Testing Python Packages

prompture

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

muteval

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

promptseal

Regression testing for prompts, agents, and models. Know what breaks before you switch.

omegaprompt

The overfit gate for your prompts: re-test the winning prompt on held-out examples it never tuned on, and block the ship if it doesn't generalize. Sits on top of promptfoo/DSPy. CLI + MCP + CI.

flowprompt-ai

Statistically sound A/B tests for prompts and models: paired tests, confidence intervals, sample-size planning and a CI regression gate. Type-safe prompts with Pydantic.

maia-test-framework

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

ai-stability

Measure LLM output consistency from the command line.

promptforge-llmops

Open-source LLMOps framework for prompt versioning, evaluation and regression testing in CI/CD pipelines.

tenro

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

replayd

Turn failed AI agent runs into replayable regression tests. Catch regressions before you ship.

mseep-agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

promptdelta

Semantic diff for LLM prompts — compare prompt versions like git diff

prompt-playoff

Prompt optimization and benchmarking for local LLMs: picks the right prompt technique for your task and measures it on Ollama. DSPy-compatible.

langalf

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

prompt-run

curl for prompts. Run .prompt files against any LLM (Anthropic, OpenAI, Ollama) from the terminal. Treat prompts as code — version them, review them in PRs, and test them in CI.