Inter Rater Reliability Python Packages

evalaudit

Statistical validity checks for human-graded AI evaluations

judgecheck

Validate LLM-as-judge reliability with inter-rater agreement, not accuracy. Reproduces a TypeScript reference implementation on a real 7-model panel.

confusion-mapper

AI-assisted distractor classification and Cohen's Kappa reliability analysis tool for cognitive science and education research.

metajudge

A reliability and DIF report card for LLM-judge and human-rater scoring instruments.

raterkit

Reliability, rogue-rater, drift and leakage diagnostics for labelled data — on the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.

kappaiou

Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"

judgepanel

Estimate LLM-judge accuracy without gold labels: Dawid-Skene EM, agreement statistics, bootstrap uncertainty