Inter Annotator Agreement Python Packages

krippendorff

Fast computation of Krippendorff's alpha agreement measure in Python.

75K 158 18

crowdtruth

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

464 62 11

iaa-kit

Modern inter-annotator agreement metrics with bootstrap intervals, ordinal support, and missing-data handling.

408 0 0

synthetic-disagreement

Synthetic reviewer disagreement generators for testing IAA and adjudication workflows.

318 0 0

bratiaa

Inter-annotator agreement for Brat annotation projects

291 22 7

crowdtruth-fork

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

228 62 11