Llm As Judge Python Packages

modaic

Python SDK for Modaic

tracely-ai

Tracely Python SDK: automatic + manual agent tracing, exported to Tracely over OTLP.

pairjudge

Pairwise LLM judges (A/B/tie): budget-aware multi-turn packing, position-bias correction, pseudo-label distillation. Generalized from the 4th-place (gold) solution to Kaggle LMSYS Chatbot Arena.

cje-eval

Causal Judge Evaluation: calibrate LLM-as-judge scores against oracle labels with valid uncertainty.

multivon-eval

Practical LLM evaluation for teams that ship to production. Deterministic + LLM-as-judge evaluators, dataset support, CI/CD integration.

modaic-client

Python SDK for Modaic

proofrag

Point your agent at your docs and your RAG app; get a golden test set + an LLM-as-judge & retrieval scorecard, in one command.

nautilus-compass

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

evalaudit

Statistical validity checks for human-graded AI evaluations

hermes-rubric

Evidence-first LLM-as-judge scoring for AI artifacts — papers, PRs, prompts, cold emails: synthesizes a rubric, collects quoted-evidence citations, scores only against that evidence, and hedges on thin evidence. Every dimension ties to a file:line or quote, with reproducibility receipts. 7 backends.

pairsort

pairsort: rank anything with AI judges — many small pairwise questions, coupled into one calibrated ranking (PKPD / Bradley–Terry) on Jev-style judges

juryrig

Audit LLM judges before you trust them — position/verbosity bias, agreement matrix, panels, calibration. Zero dependencies. PyPI: juryrig.

jev-trust

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

tracelens

Evaluation and regression-testing framework for AI agents, published in pypi.org/project/tracelens/

judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

pytest-judgekeeper

Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.

omegaprompt

The overfit gate for your prompts: re-test the winning prompt on held-out examples it never tuned on, and block the ship if it doesn't generalize. Sits on top of promptfoo/DSPy. CLI + MCP + CI.

agentsynth-ai

Outcome-verified agent trajectories, benchmarks, and RL environments — with a live leaderboard and a CI gate for your agents. Offline-first, MIT.

agentevals-evaluator-sdk

agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces

agent-report-card

A performance review for your AI agent: one CLI, one YAML test file, one markdown report. Local judge that grades itself, no API keys, every README number asserted by CI.

mini-omega-lock

Measure your LLM judge's noise floor — an A/B improvement smaller than the judge's own noise isn't real. Pre-flight probes for prompt-eval setups; works standalone and with omegaprompt.

assay-verify

Open memory & reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) — by Nautilus Platform

whatitdid

See what an AI agent actually did, not just its score. Labels every step of a CLI agent's run and compares two rounds against the noise a re-run produces anyway.

trajectory-judge

How much an LLM judge misses when an agent reaches the right answer the wrong way: a synthetic tool-using environment with injected faults, and a controlled comparison of outcome-only, step-rubric and self-consistency judges against known ground truth.