Prompt Testing Python Packages

prompture

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

agentic-security

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

ai-stability

Measure LLM output consistency from the command line.

tenro

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

maia-test-framework

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

promptforge-llmops

Open-source LLMOps framework for prompt versioning, evaluation and regression testing in CI/CD pipelines.

mseep-agentic-security

Agentic LLM vulnerability scanner

langalf

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

prompt-run

curl for prompts — run .prompt files against any LLM provider