Scraper Python Packages

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

firecrawl-py

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

zenrows

SDK to access Zenrows API directly from Python. We handle proxies rotation, headless browsers and CAPTCHAs for you.

autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

wreq

An ergonomic, privacy-aware Python HTTP Client

kameleo-local-api-client

Anti-detect browser for web scraping and automation. Engine-level fingerprint masking for Chromium and Firefox. Self-hosted, Docker-ready. Integrates with Selenium, Playwright, and Puppeteer via SDKs in Python, JavaScript, and C#.

twikit

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

rnet

An ergonomic, privacy-aware Python HTTP Client

twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

scweet

Scrape tweets, profiles, followers and following from Twitter/X, no API key needed. Python library with smart multi-account pooling, proxy support and async.

scrapetube

A YouTube scraper for scraping channels, playlists, and searching 🔎

pytrends-modern

The Modern Google Trends API - Combining the best features from pytrends, trendspyg, and more.

outscraper

The library provides convenient access to the Outscraper API from applications written in the Python language. Allows using Outscraper's services from your code.

snscrape

A social networking service scraper in Python

cinemagoer

Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people and characters

homeharvest

Python package for scraping real estate property data

twifork

Twitter & X API scraper for Python — no API key. A maintained, working fork of twikit · Twitter/X bot · internal API · free · async