Llm Judge Python Packages

llm-panel

Ask several LLMs independently, read every answer in full — parallel judges, anonymized rebuttal round, cost accounting, and a measured recall benchmark

btx-skill-jev-judge

Claude Code plugin and Python CLI: hand a judgment repeated over many items to TypeSafe's Jev, with redaction, rate limiting, a pilot and a flat-question check.

hermes-rubric

Evidence-first LLM-as-judge scoring for AI artifacts — papers, PRs, prompts, cold emails: synthesizes a rubric, collects quoted-evidence citations, scores only against that evidence, and hedges on thin evidence. Every dimension ties to a file:line or quote, with reproducibility receipts. 7 backends.

judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

pytest-judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

journeyman-bench

A process-quality benchmark for LLM agents: it grades how an agent worked, not whether it passed — and every LLM-as-a-judge sits a calibrated exam before it may score.

verdict

Inference-time scaling for LLMs-as-a-judge.

slop-eval-cli

CLI (npm + PyPI) and GitHub Action that scores AI-generated UI for genericness using an Anthropic LLM-judge rubric, with a --json mode and library API for CI and agent quality gates

clef-evals

Calibration-first evaluation toolkit for Cloudflare's Clef decision models. Judge cheap, audit confidence.

structai

StructAI offers a robust toolkit for LLM interaction—such as structured outputs, context management, and parallel execution.

fieldtest

LLM evaluation framework — define what correct, well-formed, and safe means before you measure

rewardharness

Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-model training. arXiv 2605.08703.

truescore

Statistics for LLM-judged evaluations: bias-corrected scores, real confidence intervals, and comparisons that hold up

eval-harness

A boring, config-driven harness for evaluating AI systems. One YAML drives the run, the trace is the source of truth. Offline, backtesting, and online-eval modes — works with any agent, RAG, or code-modifying system.

jedi-board-of-directors

Ask one question, get answers from several free models at different companies — they rank each other blind and a chair that didn't vote writes the decision. Runs on OpenRouter's free tier. No dependencies.

vendirag

QVS-RAG