Web Scraping Python Packages

curl-cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

primp

HTTP client that can impersonate web browsers

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

firecrawl-py

🔥 Search, scrape, and clean the web for AI agents.

patchright

Undetected Python version of the Playwright testing and automation library.

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

seleniumbase

APIs for browser automation, testing, and bypassing bot-detection.

google-search-results

Google Search Results via SERP API pip Python Package

firecrawl

🔥 Search, scrape, and clean the web for AI agents.

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

rnet

An ergonomic Python HTTP Client with TLS fingerprint

cloakbrowser

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

finvizfinance

Finviz analysis python library.

rebrowser-playwright

A drop-in replacement for playwright-python patched with rebrowser-patches. It allows to pass modern automation detection tests.

steel-sdk

The official Python library for the Steel API

abx-plugins

🧩 Plugins and extractors that ArchiveBox + abx-dl use: chrome, ytdlp, wget, singlefile, readability, forum-dl, gallery-dl, papers-dl, and more...

wreq

An ergonomic Python HTTP Client with TLS fingerprint

pyreqwest-impersonate

HTTP client that can impersonate web browsers