Evaluation Metrics Python Packages

deepeval

The LLM Evaluation Framework

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

fast-bss-eval

A fast implementation of bss_eval metrics for blind source separation

cd-fvd

[CVPR 2024] On the Content Bias in Fréchet Video Distance

codebleu

Pip compatible CodeBLEU metric implementation available for linux/macos/win

unbabel-comet

A Neural Framework for MT Evaluation

hotcoco

Perception evaluation toolkit in Rust with Python bindings. A pycocotools drop-in for COCO, LVIS, and Open Images detection metrics, up to 36× faster, plus TIDE error analysis, confusion matrices, calibration, model comparison, and a dataset browser.

pyrouge

A Python wrapper for the ROUGE summarization evaluation package

nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

calibrax

JAX benchmarking, profiling and evaluation metrics for Flax NNX: a registry of pure-function metrics (regression, classification, calibration, uncertainty, forecasting, generative, image, text, audio, graph, fairness), XLA FLOP counting, roofline analysis, GPU and energy monitoring, regression detection, publication exports, W&B and MLflow.

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

permetrics

Artificial intelligence (AI, ML, DL) performance metrics implemented in Python

vindex

Evals for Indian-language LLM apps: script adherence, mistranslation checks, calibrated similarity and an Indic LLM judge

alignmenter

Check if your AI sounds like your brand, stays safe, and behaves consistently. Works with your custom GPTs, hosted APIs, and local models. Get detailed reports in minutes, not days.

survivaleval

The most comprehensive Python package for evaluating survival analysis models.

hintauc

Hint-AUC (IEEE TVCG 2026): deterministic region-based hint generation, Hint-AUC evaluation and the DetFill colorization model — code, checkpoints, data and reproduction package

ir-metrics

The most common information retrieval (IR) metrics

pynlpl

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

nlptutti

한국어 STT 출력의 CER, WER, CRR, 키워드·개체명·코퍼스 평가를 제공하는 Python 패키지