Evaluation Python Packages

langsmith

LangSmith Client SDK Implementations

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

ragas

Supercharge Your LLM Application Evaluations 🚀

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

evo

Python package for the evaluation of odometry and SLAM

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

pytrec-eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

vincio

The context engineering platform for AI applications — compile prompts, memory, retrieval, tools, schemas & policies into optimized, validated, observable context packets.

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

arthur-observability-sdk

Make AI work for Everyone - Monitoring and governing for your AI/ML

prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

evalica

Evalica, your favourite evaluation toolkit

subset2evaluate

Find informative examples to efficiently (human)-evaluate NLG models.