cohens-kappa
Check your LLM-as-a-judge: see how often it agrees with human labels, in one command.
Validate LLM-as-judge reliability with inter-rater agreement, not accuracy. Reproduces a TypeScript reference implementation on a real 7-model panel.
Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"
