Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

lm-eval

A framework for few-shot evaluation of language models.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

scandeval

The robust European language model benchmark.

irspack

Train, evaluate, and optimize implicit feedback-based recommender systems.

cje-eval

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

euroeval

The robust European language model benchmark.

letta-evals

Evaluation kit for testing stateful agents

kiln-ai

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

retrieval-observatory

pypi retrieval reliability platform for rag pipelines: per-stage benchmarks, failure labels, stress tests, prod traces, and regression gates.

python-flexeval

FlexEval is an LLM evaluation tool designed for practical quantitative analysis.

agentlab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

aiverify-moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

ragret

Lightweight evaluation framework for Retrieval Augmented Generation systems, focused on simplicity and long-term consistency.

kiln-server

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

agent-belt

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox.

gval

A high-level Python framework to evaluate the skill of geospatial datasets by comparing candidates to benchmark maps producing agreement maps and metrics.

kaiko-eva

Evaluation framework for oncology foundation models (FMs)

modelradar

Aspect-based Forecasting Accuracy

hnep

Hybrid Network Evaluation Protocol — multi-method evaluation for hybrid quantum-classical ML models. Classifies your quantum component as Genuine, Regularizer, Ignored, or Dead Weight with bootstrap confidence intervals.

clean-evals

Try out your prompts and context across AI models. Run evals and find the best model for your use case.

zenoml-image-classification

AI Data Management & Evaluation Platform

aiterate

AI artifact lifecycle management for prompts and agent skills: optimize, evaluate, version, trace, and promote changes from raw data and policies.