Evaluation Python Packages

langsmith

LangSmith Client SDK Implementations

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

ragas

Supercharge Your LLM Application Evaluations 🚀

mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

evo

Python package for the evaluation of odometry and SLAM

nemo-gym

Evaluate and improve models and agents using environments

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

pytrec-eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

reseval

Reproducible Subjective Evaluation

evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

codebleu

Pip compatible CodeBLEU metric implementation available for linux/macos/win

vernier

A modern toolkit to evaluate 2D detection and segmentation models

hue-run

Hue TypeScript and Python SDKs for OpenTelemetry tracing and local evaluations

evalidate

Safe and fast evaluation of untrusted user-supplied python expressions

whileai

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

zeroproof

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

subset2evaluate

Find informative examples to efficiently (human)-evaluate NLG models.

evaluatorq

Evaluation module for the orq platfrom in python, including Red Teaming and Agent Simulation