Llm Benchmark Python Packages

gauntlet-cli

Behavioral reliability under pressure. Test how LLMs behave when things get hard.

workswithagents

Works With Agents — Agent OSI Model Python SDK

context-bench

Benchmark any system that transforms LLM context: compressors, RAG rerankers, memory managers, and more.

arguslm

The hundred-eyed watcher for your LLM providers. Monitor uptime, TTFT, TPS, and latency across OpenAI, Anthropic, Azure, Bedrock, Ollama, LM Studio, and 100+ providers through a single dashboard. Benchmark, compare, and get alerts — all self-hosted.

pickyourllm

Pick Your LLM: Intelligent, Use-Case Aware LLM Model advisor for Optimal Performance and Cost