Crawling Python Packages

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

spidermon

Scrapy Extension for monitoring spiders execution.

arxiv2text

Converting PDF files to text, mainly with a focus on arXiv papers.

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

grab

Web Scraping Framework

newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

scrapyrt

HTTP API for Scrapy spiders

memorious

A minimalistic, recursive web crawling library for Python.

mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

aioscpy

An asyncio + aiolibs crawler imitate scrapy framework

crawlbase

Fast python library for the Crawlbase API

proxycrawl

ProxyCrawl Python library for scraping and crawling

shopscout

Scrape any Shopify store - products, collections, pages, metadata & reviews from the public JSON API. SDK + CLI + REST API.

proxybroker2

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

maalfrid-toolkit

Toolkit for the Målfrid project

lulu

[Unmaintained] A simple and clean video/music/image downloader 👾

spidy-web-crawler

The simple, easy to use command line web crawler.

aminer-scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.