Evaluation Metrics Python Packages

deepeval

The LLM Evaluation Framework

faster-coco-eval

Continuation of an abandoned project fast-coco-eval

agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

unitxt

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

unbabel-comet

A Neural Framework for MT Evaluation

nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13

permetrics

Artificial intelligence (AI, ML, DL) performance metrics implemented in Python

fast-bss-eval

A fast implementation of bss_eval metrics for blind source separation

cd-fvd

[CVPR 2024] On the Content Bias in Fréchet Video Distance

prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

pyrouge

A Python wrapper for the ROUGE summarization evaluation package

codebleu

Pip compatible CodeBLEU metric implementation available for linux/macos/win

rliable

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

octis

OCTIS: Comparing Topic Models is Simple! A python package to optimize and evaluate topic models (accepted at EACL2021 demo track)

pynlpl

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

classeval

Evaluation of supervised predictions for two-class and multi-class classifiers

ir-metrics

The most common information retrieval (IR) metrics

survivaleval

The most comprehensive Python package for evaluating survival analysis models.

nlptutti

STT 한글 문장 인식기 출력 스크립트의 외자 오류율(CER), 단어 오류율(WER)을 계산하는 Python 함수 패키지

ecp-runtime

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

ecp-sdk

ECP is a standardized interface for orchestrating, auditing, and enforcing authority limits in AI Agent evaluations. It moves evaluation from "brittle Python scripts" to a deterministic infrastructure protocol

rag-eval-lite

Lightweight python package for RAGs classical Information Retrieval (IR) metrics.