Evaluation Python Packages

langsmith

LangSmith Client SDK Implementations

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

ragas

Supercharge Your LLM Application Evaluations 🚀

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

evo

Python package for the evaluation of odometry and SLAM

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

pytrec-eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

opik-optimizer

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

runhouse

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

subset2evaluate

Find informative examples to efficiently (human)-evaluate NLG models.

evalica

Evalica, your favourite evaluation toolkit

fore

The fore client package

pycm

Multi-class confusion matrix library in Python