PyRank

Crawling Python Packages

Python packages with the GitHub topic crawling. Sorted by monthly downloads, with stars and forks.
adbar
courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

13.4M 177 15
scrapy
scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

4.4M 65K 12K
D4Vinci
scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

929K 87K 9K
apify
crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

849K 10K 808
codelucas
newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

456K 15K 2K
ArchiveBox
abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

314K 146 9
scrapinghub
spidermon

Scrapy Extension for monitoring spiders execution.

25K 562 100
clemfromspace
scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

17K 952 345
xberg-io
crawlberg

High-performance web crawling engine with bindings for 11 languages

11K 185 29
xberg-io
crawlberg-hermes-plugin

High-performance web crawling engine with bindings for 11 languages

3K 185 29
kreuzberg-dev
kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

3K 185 29
scrapinghub
scrapyrt

HTTP API for Scrapy spiders

2K 884 160
codelucas
newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

2K 15K 2K
alephdata
memorious

Lightweight web scraping toolkit for documents and structured data.

2K 316 64
lorey
mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

2K 1K 92
crawlbase-source
crawlbase

Fast python library for the Crawlbase API

2K 26 3
lorien
grab

Web Scraping Framework

2K 2K 272
dsdanielpark
arxiv2text

Converting PDF files to text, mainly with a focus on arXiv papers.

2K 25 2
proxycrawl
proxycrawl

ProxyCrawl Python library for scraping and crawling

1K 58 19
ihandmine
aioscpy

An asyncio + aiolibs crawler imitate scrapy framework

1K 114 10
ivan-sincek
bot-safe-agents

A library for fetching a list of bot-safe user agents.

621 5 0
iawia002
lulu

[Unmaintained] A simple and clean video/music/image downloader 👾

594 800 138
bluet
proxybroker2

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

526 1K 139
scrapy
aminer-scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

453 65K 12K