Web Crawling Python Packages

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

botasaurus

The All in One Framework to Build Undefeatable Scrapers

bota

The All in One Framework to Build Undefeatable Scrapers

javascript-fixes

The All in One Framework to Build Undefeatable Scrapers

botasaurus-humancursor

The All in One Framework to Build Undefeatable Scrapers

botasaurus-server

The All in One Framework to Build Undefeatable Scrapers

bose

The Ultimate Web Scraping Framework

genmine

GenBank Record downloader for taxonomists

microwler

A micro-framework for asynchronous deep crawls and web scraping with Python

pricetag

Extract price and currency information from unstructured text. Pure Python library for e-commerce data extraction and web scraping.

nightcrawler-mitm

A python program that crawls a website and tries to stress it, polluting forms with bogus data

agentsearchcli

Give any AI agent the ability to search, crawl, and extract the web.

webtranspose

Web scraping API for building AI applications.

katastrophe

Command Line Tool to download torrents

thordata-firecrawl

Thordata Crawl – Turn any website into AI-ready data with a single API.

insite

A lightning fast tool for crawling websites and compiling PDFs of their pages

iflow-mcp-sadiuysal-crawl4ai-mcp-server

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

scrapy-googlechat

Send crawl reports from Scrapy spiders to Google Chat

pg-cache-storage

The All in One Framework to Build Undefeatable Scrapers

astel

An asyncronous web crawling library for Python.

seen

Supported JavaScript Web crawling framework for everyone.

new-frontera

A scalable frontier for web crawlers

sqlite-cache-storage

The All in One Framework to Build Undefeatable Scrapers