Common Crawl Python Packages

dingo-python

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

ccdown

A rust based, resumable downloader cli and python library for Common Crawl data

distributed-curator

Partition-aware MinHash LSH deduplication library for large-scale text data curation on Apache Spark.

websieve

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

gau-python

This library gets urls from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl. Inspired by Corbin Leo's gau

dingo-client

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool