Llm Observability Python Packages

logfire

AI observability platform for production LLM and agent systems.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

openlit

OpenLIT is the open-source agent harness engineering platform: trace, evaluate, guard, and improve everything around the model in your AI agents, on OTEL.

clawmetry

See your agent think. Zero-config observability & governance for 30 AI agent runtimes: Claude Code, OpenAI Codex, Hermes, OpenClaw & 26 more. Live token costs, sessions, tool calls, crons.

judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

my-claude-code

My Claude Code (MCC) is a local multi-provider LLM proxy, model router and control plane for Claude Code, Codex, OpenCode, Gemini CLI and other AI coding agents — 57 providers, fallback routing, credential health, a native web-search tool proxy, and exhaustive analytics with cost provenance in one local dashboard.

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

helicone-helpers

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

latitude-sdk

Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.

atflows

AtFlows is a local observability tool for LLM applications. Point your SDK at it, see your costs, tokens, and latency in real-time.

latitude-telemetry

Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.

isnad

Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines

opik-mcp

Model Context Protocol (MCP) server for Opik, the open-source LLM observability and evaluation platform, built by Comet. Read traces, log scores, and manage prompts from Claude Code, Cursor, or VS Code.

argus-obs-semconv

Production-grade observability and agentic AIOps for AI systems — OpenTelemetry-native across independent polyglot apps; GenAI semantic conventions with token/cost attribution; sub-2s detection on a hot path that skips the datastore; LLM agents that find root cause and notify. ClickHouse + Langfuse. 100% self-hosted — no vendor lock-in.

argus-obs-sdk

Production-grade observability and agentic AIOps for AI systems — OpenTelemetry-native across independent polyglot apps; GenAI semantic conventions with token/cost attribution; sub-2s detection on a hot path that skips the datastore; LLM agents that find root cause and notify. ClickHouse + Langfuse. 100% self-hosted — no vendor lock-in.

argus-obs-schemas

Production-grade observability and agentic AIOps for AI systems — OpenTelemetry-native across independent polyglot apps; GenAI semantic conventions with token/cost attribution; sub-2s detection on a hot path that skips the datastore; LLM agents that find root cause and notify. ClickHouse + Langfuse. 100% self-hosted — no vendor lock-in.

genai-otel-instrument

GenAI OpenTelemetry Auto-Instrumentation Library A comprehensive wrapper for automatic instrumentation of LLM/GenAI applications Supports all major LLM providers and MCP (Model Context Protocol) tool calls

tracely-ai

Tracely Python SDK: automatic + manual agent tracing, exported to Tracely over OTLP.

shyftlabs-continuum

Continuum — the agent runtime by ShyftLabs. Build, orchestrate, ship.

alignmenter

Check if your AI sounds like your brand, stays safe, and behaves consistently. Works with your custom GPTs, hosted APIs, and local models. Get detailed reports in minutes, not days.

agentic-ledger

Local-first flight recorder for AI agents: every LLM call, tool call, cost, and loop captured by a transparent proxy. Zero code changes.

cognifity-verdict

Open-source LLM observability and drift detection for AI apps

cognifity-verdict-eval

Open-source LLM observability and drift detection for AI apps

pf-core

Python foundation for LLM apps whose prompts and spend you can actually see — versioned prompts, every call recorded and replayable, budgets, evals, jobs.