Llm Benchmark Python Packages

trainlens

TrainLens turns training runs into evidence: what changed, what went wrong, and what experiment to run next.

calibrate-sdk

Python SDK for Calibrate's API

haenv

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

make-localllm-easier

One command runs the best local LLM your GPU can handle: model picked from measured multilingual benchmarks, llama.cpp tuned for AMD, NVIDIA, Intel and Apple GPUs.

llm-speed

Measure local LLM speed, first-token latency, and throughput. Reproducible benchmarks across GPUs and Apple Silicon.

gauntlet-cli

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

trustllm

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

pickyourllm

Pick Your LLM: Intelligent, Use-Case Aware LLM Model advisor for Optimal Performance and Cost

glossobench

Open-source, judge-agnostic LLM benchmark for any language — natural or AI-developed. Turn hours to minutes, save days of benchmark cost! Public data, no gate, $0 offline evaluation.

arguslm

The hundred-eyed watcher for your LLM providers. Monitor uptime, TTFT, TPS, and latency across OpenAI, Anthropic, Azure, Bedrock, Ollama, LM Studio, and 100+ providers through a single dashboard. Benchmark, compare, and get alerts — all self-hosted.

prompt-playoff

Prompt optimization and benchmarking for local LLMs: picks the right prompt technique for your task and measures it on Ollama. DSPy-compatible.

memorylens-bench

The open-source benchmark for LLM memory decay. Measure how Naive, RAG, Chunked RAG, Cascading, and SummaryMemory degrade over 100 conversation turns. Ebbinghaus forgetting curves, 5-provider LLM eval, multi-seed CI. No API key needed.

context-bench

Benchmark any system that transforms LLM context: compressors, RAG rerankers, memory managers, and more.