Crawler Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

13.6M 7K 455

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

13.4M 177 15

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

5.6M 2K 94

firecrawl-py

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

5.5M 190K 10K

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

4.4M 65K 12K

crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

1.3M 86K 9K

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

929K 87K 9K

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

849K 10K 808

google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

727K 1K 250

firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

716K 190K 10K

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

456K 15K 2K

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

443K 26 9

random-user-agent

A package to get list of user agents based on filters such as operating system, software name etc..

299K 102 12

autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

218K 8K 831

wreq

An ergonomic, privacy-aware Python HTTP Client

206K 1K 123

kameleo-local-api-client

Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.

161K 212 30

rnet

An ergonomic, privacy-aware Python HTTP Client

102K 1K 123