Crowdsourcing Python Packages

weblate

Web based localization tool with tight version control integration.

crowd-kit

Control the quality of your labeled data with the Python tools you already know.

toloka-kit

Toloka-Kit is a Python library for working with Toloka API.

wq

📱🌐📋 wq: a modular framework supporting web / native apps for mobile surveys and geospatial data collection. Powered by Django REST Framework, Redux, React, and Material UI.

crowdtruth

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

crowdnalysis

Library to help analyze crowdsourcing results

weblate-a1az

Web based localization tool with tight version control integration.

weblate-em0t

Web based localization tool with tight version control integration.

raterkit

Reliability, rogue-rater, drift and leakage diagnostics for labelled data — on the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.

garner

A library containing all you need to easily integrate with the Garner data crowdsourcing system

judgepanel

Estimate LLM-judge accuracy without gold labels: Dawid-Skene EM, agreement statistics, bootstrap uncertainty

slopit

Tools for detecting the use of AI assistance in crowd-sourced data collection

turkle

Django-based clone of Amazon's Mechanical Turk service running in your local environment.

crowded

🤖 Guideline for designing optimal crowdsourcing experiments

crowdtruth-fork

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems