Web Scraping Python Packages

curl-cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

primp

HTTP client that can impersonate web browser tls/http2 fingerprints

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

firecrawl-py

The API to search, scrape, and interact with the web at scale. 🔥

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

patchright

Undetected Python version of the Playwright testing and automation library.

seleniumbase

📊 APIs for web automation, testing, and bypassing bot-detection.

apify-client

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

google-search-results

Google Search Results via SERP API pip Python Package

rnet

An ergonomic, censorship-resistant Python HTTP Client

firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

httpmorph

httpmorph is a drop-in replacement for Python's requests library that uses a custom C implementation with BoringSSL instead of Python's standard HTTP stack.

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

cloakbrowser

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

apify

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

finvizfinance

Finviz analysis python library.

steel-sdk

The official Python library for the Steel API

rebrowser-playwright

A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.