Llm Benchmarking Python Packages

elastik

AuditeDB: the db that listens. Python client for the Elastik L5 Engine.

openvals

Open-source AI model evaluation and benchmarking framework for LLMs (OpenAI, Ollama, Claude, Gemini)

awb

Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.

agent-action-guard

🛡️ Safe AI Agents through Action Classifier

porchbench

Rigorous quality benchmarking for local LLMs - paired stats, LLM-as-judge scoring, routing discovery, and reproducible runs on your hardware.

evalsense

Tools for systematic large language model evaluations

trusteval-ai

Enterprise LLM Evaluation & Responsible AI Framework — Benchmark bias, hallucination, PII leakage, and toxicity across Healthcare, BFSI, Retail & Legal industries. Supports OpenAI, Anthropic, Gemini & HuggingFace. Python SDK + CLI + Web Dashboard. 191 tests. Compliance-ready reports.