Llm Benchmark Python Packages

gauntlet-cli

Community-driven behavioral reliability benchmark for LLMs. 231 probes across 19 modules, deterministic scoring, perplexity correlation, layer sensitivity mapping, quant method capture, hardware-stratified community rankings. Every test contributes to the community dataset.

context-bench

Benchmark any system that transforms LLM context: compressors, RAG rerankers, memory managers, and more.

pickyourllm

Pick Your LLM: Intelligent, Use-Case Aware LLM Model advisor for Optimal Performance and Cost

arguslm

ArgusLM — Open-source LLM monitoring & benchmarking SDK