Evals Python Packages

logfire

AI observability platform for production LLM and agent systems.

arize-phoenix

AI Observability & Evaluation

arize-phoenix-otel

AI Observability & Evaluation

arize-phoenix-client

AI Observability & Evaluation

arize-phoenix-evals

AI Observability & Evaluation

agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

trulens-core

Evaluation and Tracking for LLM Experiments and AI Agents

trulens

Evaluation and Tracking for LLM Experiments and AI Agents

harbor-rewardkit

Harbor is a framework for running agent evaluations and creating and using RL environments.

trulens-providers-litellm

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-dashboard

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-feedback

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-otel-semconv

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-eval

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-connectors-snowflake

Evaluation and Tracking for LLM Experiments and AI Agents

shadow-diff

Behavior contracts for AI agents

evalica

Evalica, your favourite evaluation toolkit

trulens-providers-cortex

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-providers-openai

Evaluation and Tracking for LLM Experiments and AI Agents

trulens-apps-langchain

Evaluation and Tracking for LLM Experiments and AI Agents

mcp-assert

The deterministic testing standard for MCP servers. Connect over real stdio/SSE/HTTP transport, call tools with real arguments, assert results with 18 assertion types defined in YAML. Any language, any transport, no mocks. Single Go binary.

trulens-apps-llamaindex

Evaluation and Tracking for LLM Experiments and AI Agents

cellin

Build long-lived multimodal memory, dream over it, and retrieve context with transparent weighting.

selectools

Production-ready Python framework for AI agents with built-in guardrails, audit logging, cost tracking, and hybrid RAG. Supports OpenAI, Anthropic, Gemini, Ollama. By NichevLabs.