Scraping Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

fake-useragent

Up-to-date simple useragent faker with real world database

firecrawl-py

The API to search, scrape, and interact with the web at scale. 🔥

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

browserforge

🎭 Intelligent browser header & fingerprint generator

apify-client

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

camoufox

🦊 Anti-detect browser

google-search-results

Google Search Results via SERP API pip Python Package

apify-fingerprint-datapoints

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

fake-http-header

A python package to generate random request fields for a http header.

requests-html

Pythonic HTML Parsing for Humans™

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

apify

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

zenrows

SDK to access ZenRows API directly from Python. We handle proxies rotation, headless browsers and CAPTCHAs for you.

twikit

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

socid-extractor

⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites

oxylabs-mcp

Official Oxylabs MCP integration

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

shot-scraper

A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript