Judge Python Packages

openevalkit

Production-grade Python framework for evaluating LLM and agentic systems with traditional scorers, LLM judges (OpenAI, Anthropic, Ollama, 100+ models via LiteLLM), ensemble aggregation, and smart caching for cost-effective testing.

cyaron

CYaRon: Yet Another Random Olympic-iNformatics test data generator

boj

BOJ-Offline-Judge는 백준 온라인 저지를 CLI, 혹은 Python 스크립트를 통해 이용 하기 위해 제작한 API 입니다.

local-judge

judge the program locally

judge-bench

Probe judge behavior for position, verbosity, self-preference, paraphrase, anchoring, and calibration failures.

judge-card

Validate and render a disclosure record for judge prompts, calibration, bias, limitations, and intended use.

judie

The chunking benchmark

mtcb

The chunking benchmark