Crowdsourcing Python Packages

weblate

Web based localization tool with tight version control integration.

crowd-kit

Control the quality of your labeled data with the Python tools you already know.

toloka-kit

Toloka-Kit is a Python library for working with Toloka API.

wq

A modular framework for mobile surveys and field data collection via offline-capable mobile web apps.

crowdtruth

Disagreement based metrics for the processing and evaluation of crowdsourced annotations

crowdnalysis

Library to help analyze crowdsourcing results

weblate-a1az

A web-based continuous localization system with tight version control integration

weblate-em0t

A web-based continuous localization system with tight version control integration

raterkit

Reliability, rogue-rater, drift and leakage diagnostics for labelled data

garner

A library for interacting with the Garner API.

judgepanel

Estimate LLM-judge accuracy without gold labels and aggregate judge panels: Dawid-Skene EM, agreement statistics, bootstrap uncertainty

slopit

AI response detection toolkit for crowdsourced behavioral research

turkle

Django-based clone of Amazon's Mechanical Turk service

crowded

This package contains the methods to generate simulated workers and tasks of a crowdsourcing use case

crowdtruth-fork

Disagreement based metrics for the processing and evaluation of crowdsourced annotations