Llm Observability Python Packages

logfire

AI observability platform for production LLM and agent systems.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

latitude-sdk

Latitude is the open-source ai monitoring platform.

latitude-telemetry

Latitude is the open-source ai monitoring platform.

helicone-helpers

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

acontext

Agent Skills as a Memory Layer

genai-otel-instrument

GenAI OpenTelemetry Auto-Instrumentation Library A comprehensive wrapper for automatic instrumentation of LLM/GenAI applications Supports all major LLM providers and MCP (Model Context Protocol) tool calls

dunetrace

Real-time monitoring of production AI agents. No raw content transmitted.

peekr

Zero-config observability for AI agents. Auto-instruments OpenAI & Anthropic SDKs.

helicone-async

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

mcpeye

mcpeye Python SDK — open-source product analytics for MCP servers. See why your agent is failing.

comet-llm

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

shyftlabs-continuum

Continuum — the agent runtime by ShyftLabs. Build, orchestrate, ship.

agent-panorama

See what your AI agents do, whether it's worth it, and what it costs - a manager-readable report + local dashboard from Langfuse/LangSmith traces (or a one-line live callback). Open source, runs locally.

agentmetrics

Open-source AI agent observability. Track cost, latency, tokens, and errors for OpenClaw, Hermes, LangChain, CrewAI, LlamaIndex, OpenAI Agents, AutoGen, and Anthropic Managed Agents. Self-hosted, no cloud required.

forgesight-api

Vendor-neutral, OpenTelemetry-first telemetry for AI agents — traces, cost, budgets & a tamper-evident audit trail to any backend, no agent-code changes.

forgesight-core

Vendor-neutral, OpenTelemetry-first telemetry for AI agents — traces, cost, budgets & a tamper-evident audit trail to any backend, no agent-code changes.

helicone

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

agentmetrics-hermes

Open-source AI agent observability. Track cost, latency, tokens, and errors for OpenClaw, Hermes, LangChain, CrewAI, LlamaIndex, OpenAI Agents, AutoGen, and Anthropic Managed Agents. Self-hosted, no cloud required.

longtrainer

Production-ready RAG framework for Python — multi-tenant chatbots with streaming, tool calling, agent mode (LangGraph), vector search (FAISS), and persistent MongoDB memory. Built on LangChain.

spanlens

Spanlens SDK for Python. Agent tracing, LLM usage capture, and cost observability.