krippendorff-alpha
Statistical validity checks for human-graded AI evaluations
Validate LLM-as-judge reliability with inter-rater agreement, not accuracy. Reproduces a TypeScript reference implementation on a real 7-model panel.
A reliability and DIF report card for LLM-judge and human-rater scoring instruments.
Reliability, rogue-rater, drift and leakage diagnostics for labelled data — on the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.
Analyse the qualitative coding exported by the zotQDA and qdaZ Zotero plugins, or from REFI-QDA projects
