Prompt Evaluation Python Packages

compare-prompts

A lightweight CLI tool for evaluating LLM prompts. Run prompts side-by-side to instantly compare token usage, tone, reading level, and API costs. Natively supports OpenAI, Anthropic, Gemini, Groq, and Ollama. Perfect for safely refactoring prompts, tuning bot personalities, or proving that "be concise" actually lowers your bill.

promptstats

Statistical analysis methods for comparing prompt and model performance in LLM evaluations.

judicator

Judging LLM-as-a-Judge — a screening tool for bias and miscalibration.

evalstats

Statistical analysis methods for comparing prompt and model performance in LLM evaluations.

prompt-foundry-python-sdk

The prompt engineering, prompt management, and prompt evaluation tool for Python