Llm As A Judge Python Packages

whileai

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

zeroproof

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

llm-council-core

The LLM Council works together to answer your hardest questions

oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

jev-judge-mcp

Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.

grounded-ai

Evals that meet you where you are. For AI that's grounded.

openevalkit

Production-grade Python framework for evaluating LLM and agentic systems with traditional scorers, LLM judges (OpenAI, Anthropic, Ollama, 100+ models via LiteLLM), ensemble aggregation, and smart caching for cost-effective testing.

llmswap

Ask, compare, and synthesize across 10 LLM providers. Python SDK, CLI, local Arena, MCP, and privacy-aware

semgate

Catches your AI agent doing things you never asked it to do. Built for OpenCode, Pi, GitHub Copilot CLI, oh-my-pi and Antigravity CLI.

ailens-evals

Evals for AI apps that make text, images, video or audio: is every change better or worse, why, and what it costs. pip install ailens-evals

vllm-judge

A tiny, lightweight library for LLM-as-a-Judge evaluations on vLLM-hosted models.

ragscore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

assevra

An open-source reliability scorecard for LLM agents — measure grounding, safety, PII, and task-completion against fixed thresholds with confidence intervals.

dingo-python

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

pytest-judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

journeyman-bench

A process-quality benchmark for LLM agents: it grades how an agent worked, not whether it passed — and every LLM-as-a-judge sits a calibrated exam before it may score.

verdict

Inference-time scaling for LLMs-as-a-judge.

mcp-as-a-judge

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations

convy

Conversation tests for AI agents: a simulated user talks to your agent through its API, a judge checks every claim, and convy reports pass rate, tokens, time and what changed between builds.

docling-sdg

A set of tools to create synthetically-generated data from documents

nasde-toolkit

CLI for benchmarks & evals of AI coding agents — on tasks you already understand, using your Claude / Codex / Gemini individual subscriptions or API keys.

better-cheaper-llm

Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.

root-signals

Scorable SDK