Llm Benchmarking Python Packages

awb

Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.

agent-action-guard

🛡️ Safe AI Agents through Action Classifier

localdeploy

Find, deploy, manage, and benchmark local AI models for your hardware across Ollama, llama.cpp, and compatible runtimes.

bench-my-llm

🏎️ Dead-simple LLM benchmarking CLI - latency, cost, and quality metrics

openvals

Open-source AI model evaluation and benchmarking framework for LLMs (OpenAI, Ollama, Claude, Gemini)

evalsense

Tools for systematic large language model evaluations

llamathon

🦙 Put your LLM server through a marathon — one command to benchmark any OpenAI-compatible endpoint (Ollama, vLLM, llama.cpp), with model auto-detection, real-workload sizing, and self-contained HTML reports.

trusteval-ai

Enterprise LLM Evaluation & Responsible AI Framework — Benchmark bias, hallucination, PII leakage, and toxicity across Healthcare, BFSI, Retail & Legal industries. Supports OpenAI, Anthropic, Gemini & HuggingFace. Python SDK + CLI + Web Dashboard. 191 tests. Compliance-ready reports.

porchbench

Rigorous quality benchmarking for local LLMs - paired stats, LLM-as-judge scoring, routing discovery, and reproducible runs on your hardware.