Llm As A Judge Python Packages

agenta

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

llm-council-core

The LLM Council works together to answer your hardest questions

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

verdict

Inference-time scaling for LLMs-as-a-judge.

dingo-python

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

evret

Evals framework for Information Retrieval Systems

judicator

Who evaluates the evaluator? Judicator audits LLM-as-a-Judge systems for 7 documented bias types. Zero config. Works with any LLM.

scorable

The Python SDK for API of Scorable

nasde-toolkit

CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

llm-summary

Use an LLM to summarize paragraphs

root-signals

Scorable SDK

docling-sdg

A set of tools to create synthetically-generated data from documents

mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations

vllm-judge

A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.

pytest-llm-rubric

Pytest plugin for semantic PASS/FAIL checks using LLM-as-a-Judge

openevalkit

Open evaluation kit for LLM systems

veritail

LLM-as-a-Judge evaluation platform for ecommerce search. Scores relevance, computes IR metrics, and flags quality issues across multiple retail verticals

xfinder

[ICLR 2025] xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

artemis-agents

Production-ready multi-agent debate framework with adaptive evaluation and safety monitoring

llm-antibodies

Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)

root-signals-cli

CLI for the Root Signals API

antibodies-rafaelsandroni

Antibodies for LLMs hallucinations (grouping LLM as a judge, NLI, reward models)

scorable-cli

CLI for the Scorable API

iflow-mcp-hepivax-mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations