Evaluation Framework Python Packages

deepeval

The LLM Evaluation Framework

lm-eval

A framework for few-shot evaluation of language models.

coder-eval

Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.

scandeval

The robust European language model benchmark.

lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

irspack

Train, evaluate, and optimize implicit feedback-based recommender systems.

alignmenter

Check if your AI sounds like your brand, stays safe, and behaves consistently. Works with your custom GPTs, hosted APIs, and local models. Get detailed reports in minutes, not days.

euroeval

The robust European language model benchmark.

kiln-ai

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

cje-eval

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

decimalai

Python SDK for agent evals and skill routing — measure a skill's real lift before you trust it

juried

Acceptance testing for LLM features, with verdicts corrected for the judge's own error rate.

acruxcore

Open-source LLMOps platform with prompt management, an OpenAI-compatible AI gateway, OpenTelemetry tracing, a versioned tool catalog, and LLM evaluation. Use hosted or self-host with TypeScript and Python SDKs.

letta-evals

Evaluation kit for testing stateful agents

kiln-server

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

retrieval-observatory

pypi retrieval reliability platform: per-stage visibility, benchmarks, tracing, and integration

rvcbench

Comprehensive voice cloning evaluation: seven automatic speech metrics and ready-to-use benchmark datasets. Score any model’s audio with Python or CLI.

agentlab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

crucible-bench

Python and MCP tools for testing falsifiable claims and recording MATCH, DRIFT, or UNVERIFIABLE outcomes against supplied evidence.

aiverify-moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

kaiko-eva

Evaluation framework for oncology foundation models (FMs)

pyrddlgym

A toolkit for auto-generation of OpenAI Gym environments from RDDL description files.

uhbs

Universal Honeypot Benchmarking Standard (UHBS) v5.0.x — open-source framework for vendor-neutral honeypot / deception evaluation

agent-eval-flow

Evaluate complete agent systems: capture execution evidence, inspect reports, and compare improvements.