Crawling Python Packages

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

spidermon

Scrapy Extension for monitoring spiders execution.

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

arxiv2text

Converting PDF files to text, mainly with a focus on arXiv papers.

grab

Web Scraping Framework

proxycrawl

A Python class that acts as wrapper for ProxyCrawl scraping and crawling API

crawlberg

High-performance web crawling engine with bindings for 11 languages

crawlbase

Fast python library for the Crawlbase API

mlscraper

Scrape HTML automatically with machine learning.

newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapyrt

HTTP API for Scrapy spiders

memorious

Lightweight web scraping toolkit for documents and structured data.

aioscpy

An asyncio + aiolibs crawler imitate scrapy framework

maalfrid-toolkit

Toolkit for the Målfrid project

qcrawl

qcrawl - fast async web crawling & scraping framework for Python.

docscrape

Scrape any documentation site to Markdown in seconds

aminer-scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

lulu

A simple and clean video/music/image downloader that supports many websites ��