Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

lm-eval

A framework for few-shot evaluation of language models.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

scandeval

The robust European language model benchmark.

irspack

Train, evaluate, and optimize implicit feedback-based recommender systems.

euroeval

The robust European language model benchmark.

kiln-ai

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

letta-evals

Evaluation kit for testing stateful agents

agentlab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

aiverify-moonshot

AI Verify advances Gen AI testing with Project Moonshot.

kiln-server

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

kaiko-eva

Evaluation framework for oncology foundation models (FMs)

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

zenoml

Interactive Evaluation Framework for Machine Learning

trustlens

Open-source Python library for evaluating ML model reliability beyond accuracy — with calibration, failure, and fairness diagnostics for informed deployment decisions.

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

k-eval

Simple context-aware evaluation framework for AI agents using MCP.

modelradar

Aspect-based Forecasting Accuracy

ceval

A framework for evaluating counterfactual explanations.

zenoml-image-classification

AI Data Management & Evaluation Platform

gval

A high-level Python framework to evaluate the skill of geospatial datasets by comparing candidates to benchmark maps producing agreement maps and metrics.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

lapixdl

Python package with Deep Learning utilities for Computer Vision

nlproc-tools

🌾 Universal, customizable and deployable fine-grained evaluation for text generation.