Llm Observability Python Packages

logfire

AI observability platform for production LLM and agent systems.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

helicone-helpers

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

acontext

Agent Skills as a Memory Layer

genai-otel-instrument

GenAI OpenTelemetry Auto-Instrumentation Library A comprehensive wrapper for automatic instrumentation of LLM/GenAI applications Supports all major LLM providers and MCP (Model Context Protocol) tool calls

pathlight

Visual debugging, execution traces, and observability for AI agents.

agentminds

Python SDK for AgentMinds — cross-site collective intelligence for production AI agents. Auto-capture errors + push/pull patterns from the network pool.

dunetrace

Real-time monitoring of your production agents. No raw content transmitted.

observal-cli

Observal is an Observability and Evaluation platform for human-in-the-loop agents

helicone

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

browsertrace

Local replay debugger for Browser Use failures with screenshots, model I/O, failed-step timelines, and public-safe HTML exports.

longtrainer

Production-ready RAG framework for Python — multi-tenant chatbots with streaming, tool calling, agent mode (LangGraph), vector search (FAISS), and persistent MongoDB memory. Built on LangChain.

burnlens

Open-source LLM FinOps proxy — track OpenAI, Anthropic (Claude), and Google Gemini costs by feature, team, and customer. Zero code changes. pip install burnlens.

comet-llm

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

helicone-async

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

otel-genai-graph

Project OpenTelemetry GenAI traces into a queryable graph (Neo4j or DuckDB) — agent delegation, cost attribution, blast radius.

annexkit

EU AI Act compliance pipeline for developers — SDK, collector, Annex IV PDF generator. Open-core.

llmkit-sdk

Know what your AI agents cost. API gateway with budget enforcement, session tracking, and MCP tools.

peaky-peek-server

Self-hosted server for Agent Debugger — FastAPI backend, SQLite/Postgres storage, SSE streaming, and React UI.

peaky-peek

Lightweight tracing SDK for AI agents. Capture decisions, tool calls, and LLM events with one context manager.

ambertrace

Python SDK for tracing LLM calls (OpenAI, Anthropic, Google Gemini) to AmberTrace observability platform