Scraping Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

fake-useragent

Up-to-date simple useragent faker with real world database

firecrawl-py

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

apify-client

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

browserforge

🎭 Intelligent browser header & fingerprint generator

apify-fingerprint-datapoints

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

google-search-results

Google Search Results via SERP API pip Python Package

camoufox

🦊 Anti-detect browser

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

fake-http-header

A python package to generate random request fields for a http header.

shot-scraper

A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript

scrapy-stealth

Stealthy Crawling. Maximum Results. A pluggable anti-bot and stealth framework for Scrapy.

abx-plugins

🧩 Plugins and extractors that ArchiveBox + abx-dl use: chrome, ytdlp, wget, singlefile, readability, forum-dl, gallery-dl, papers-dl, and more...

requests-html

Pythonic HTML Parsing for Humans™

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

zenrows

SDK to access Zenrows API directly from Python. We handle proxies rotation, headless browsers and CAPTCHAs for you.

autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

apify

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction