PyRank

Inter Annotator Agreement Python Packages

Python packages with the GitHub topic inter-annotator-agreement. Sorted by monthly downloads, with stars and forks.
pln-fing-udelar
krippendorff

Fast computation of Krippendorff's alpha agreement measure in Python.

122K 161 18
CrowdTruth
crowdtruth

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

254 64 10
auraoneai
iaa-kit

Measure inter-annotator agreement with modern metrics, uncertainty, ordinal support, and missing-data handling.

199 0 0
kldtz
bratiaa

Inter-annotator agreement for Brat annotation projects

162 22 7
tr7200
kappaiou

Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"

101 0 0
auraoneai
synthetic-disagreement

Generate controlled synthetic annotator disagreement before real reviewer data is collected.

96 0 0
CrowdTruth
crowdtruth-fork

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

60 64 10