Web Crawler Python Packages

firecrawl-py

🔥 Search, scrape, and clean the web for AI agents.

6.8M 121K 7K

firecrawl

🔥 Search, scrape, and clean the web for AI agents.

971K 121K 7K

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

536K 9K 742

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

318K 76 14

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

256K 55 15

stealth-requests

Undetected web-scraping & seamless HTML parsing in Python!

45K 470 48

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

10K 97 12

search-ai-core

Search the web with advanced filters and LLM-friendly output formats!

8K 57 2

crw

Fast, lightweight Firecrawl alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/v1/scrape, /v1/crawl, /v1/search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

6K 89 5

spider-rs

Spider ported to Python

5K 107 17

redis-queue-tool

Distributed task redisqueue(最简单python分布式函数调度框架)

2K 65 19

ladon-crawl

A Python framework for building structured, resumable web crawlers — designed for domains where data quality matters.

1K 5 1

leek

Distributed task redisqueue(最简单python分布式函数调度框架)

1K 65 19

hivewebcrawler

Python 3.x Web Crawler, Images, Urls, Emails, Phone numbers

1K 3 1

wereadscan

扫描“微信读书”已购图书并下载本地PDF的爬虫

876 992 173

infomesh

🕸️ Fully decentralized P2P search engine for LLMs — no API key, no billing, forever free. MCP-native web search via Kademlia DHT + libp2p + FTS5. Production stable ✅ → pip install infomesh

857 3 0