Web Crawling Python Packages

apify-client

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

botasaurus

The All in One Framework to Build Undefeatable Scrapers

bota

The All in One Framework to Build Undefeatable Scrapers

javascript-fixes

The All in One Framework to Build Undefeatable Scrapers

botasaurus-humancursor

The All in One Framework to Build Undefeatable Scrapers

tilion-fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

botasaurus-server

The All in One Framework to Build Undefeatable Scrapers

genmine

GenBank Record downloader for taxonomists

bose

The All in One Framework to Build Undefeatable Scrapers

nightcrawler-mitm

A python program that crawls a website and tries to stress it, polluting forms with bogus data

katastrophe

Command Line Tool to download torrents

microwler

A micro-framework for asynchronous deep crawls and web scraping written in Python

agentsearchcli

Give any AI agent the ability to search, crawl, and extract the web.

webtranspose

Web scraping API for building AI applications.

thordata-firecrawl

Thordata Firecrawl – Firecrawl-compatible web crawling & scraping API built on Thordata, turning any website into AI-ready Markdown/JSON/HTML/screenshots.

pg-cache-storage

PostgreSQL cache storage for botasaurus.

pricetag

A pure-Python library for extracting price and currency information from unstructured text

insite

A lightning fast tool for crawling websites and compiling PDFs of their pages

iflow-mcp-sadiuysal-crawl4ai-mcp-server

A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents.

scrapy-googlechat

Send crawl reports from Scrapy spiders to Google Chat

astel

An asyncronous web crawling library for Python.

seen

Supported JavaScript Web crawling framework for everyone.