Evaluation Metrics Python Packages

deepeval

The LLM Evaluation Framework

agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

unbabel-comet

A Neural Framework for MT Evaluation

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13

fast-bss-eval

A fast implementation of bss_eval metrics for blind source separation

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

cd-fvd

[CVPR 2024] On the Content Bias in Fréchet Video Distance

permetrics

Artificial intelligence (AI, ML, DL) performance metrics implemented in Python

pyrouge

A Python wrapper for the ROUGE summarization evaluation package

codebleu

Pip compatible CodeBLEU metric implementation available for linux/macos/win

classeval

Evaluation of supervised predictions for two-class and multi-class classifiers

pynlpl

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

ir-metrics

The most common information retrieval (IR) metrics

octis

OCTIS: a library for Optimizing and Comparing Topic Models.

nf1

A novel approach to evaluate community detection algorithms on ground truth

survivaleval

The most comprehensive Python package for evaluating survival analysis models.

nlptutti

STT 한글 문장 인식기 출력 스크립트의 외자 오류율(CER), 단어 오류율(WER)을 계산하는 Python 함수 패키지

evret

Evals framework for Information Retrieval Systems

ecp-sdk

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol