Web Crawling Python Packages

apify-client

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

javascript-fixes

The All in One Framework to Build Undefeatable Scrapers

botasaurus

The All in One Framework to Build Undefeatable Scrapers

botasaurus-humancursor

The All in One Framework to Build Undefeatable Scrapers

bota

The All in One Framework to Build Undefeatable Scrapers

tilion-fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

botasaurus-server

The All in One Framework to Build Undefeatable Scrapers

katastrophe

Command Line Tool to download torrents

easyproxies

EasyProxies is a fast Python library and CLI tool for fetching free proxy servers from proxyscan-like APIs. It supports HTTP, HTTPS, and SOCKS proxies with advanced filtering by country, ping, and anonymity. Features an infinite proxy generator and seamless integration with the requests library for efficient web scraping and automation.

crawl4ai-gautamp8

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

pagedigest

JSON change manifests for stateful crawlers and caches. Stable v1 wire format with publisher and consumer tooling.

aegis-stack-crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

genmine

GenBank Record downloader for taxonomists

langaccess

Classifies the language access a website provides, separating the site's own writing from a translation widget's output, and returns the evidence behind each verdict.

kalika

Web archiver and crawler based on Heritrix and pywb.

bose

The All in One Framework to Build Undefeatable Scrapers

context-dev-haystack

Haystack components for Context.dev: live web search, scraping and crawling for RAG pipelines.

crawl4ai-ctf

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

web-scraping-api-sdk

Web scraping API examples in Python, Node, Go, Java, PHP, Ruby, cURL and CLI. Proxies, JS rendering, AI extraction, 9 scrapers.

iflow-mcp-sadiuysal-crawl4ai-mcp-server

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

nightcrawler-mitm

A mitmproxy-based security research sidekick for passive analysis, targeted active testing, confidence scoring and reproducible PoCs.