Llm As A Judge Python Packages

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

llm-council-core

The LLM Council works together to answer your hardest questions

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

vllm-judge

A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.

dingo-python

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations

openevalkit

Production-grade Python framework for evaluating LLM and agentic systems with traditional scorers, LLM judges (OpenAI, Anthropic, Ollama, 100+ models via LiteLLM), ensemble aggregation, and smart caching for cost-effective testing.

clean-evals

Try out your prompts and context across AI models. Run evals and find the best model for your use case.

verdict

Inference-time scaling for LLMs-as-a-judge.

scorable

Scorable SDK

evret

Evals framework for Information Retrieval Systems

docling-sdg

A set of tools to create synthetically-generated data from documents

nasde-toolkit

CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

claim-memory-graph

A memory layer that tracks evidence, claims, and decisions to make multi-turn LLM judges and reviewer agents more inspectable and stable.

root-signals

The Python SDK for API of Root Signals

metajudge

A reliability and DIF report card for LLM-judge and human-rater scoring instruments.

llm-summary

Use an LLM to summarize paragraphs.

pytest-llm-rubric

Pytest plugin for semantic PASS/FAIL checks using LLM-as-a-Judge

veritail

LLM-as-a-Judge evaluation platform for ecommerce search. Scores relevance, computes IR metrics, and flags quality issues across multiple retail verticals

artemis-agents

A production-ready framework for structured multi-agent debates with adaptive evaluation, causal reasoning, and built-in safety monitoring.

judicator

Judging LLM-as-a-Judge — a screening tool for bias and miscalibration.

xfinder

[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

root-signals-cli

CLI for the Root Signals API

llm-antibodies

Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)