Web Crawler Python Packages

firecrawl-py

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

crawl4ai

Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

firecrawl

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

scrapegraph-py

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

stealth-requests

Undetected web-scraping & seamless HTML parsing in Python!

crawlberg

High-performance web crawling engine with bindings for 11 languages

search-ai-core

Search the web with advanced filters and LLM-friendly output formats!

agentcrawl-ai

Self-hosted web scraping and Markdown extraction for AI agents

open-news-api

open-news is a Python library for news discovery and extraction. It fetches live news via DuckDuckGo, searches Google News, discovers RSS feeds, crawls websites, and extracts article text. Features filtering, deduplication, ranking, and optional JavaScript rendering. Includes Python API, CLI, and terminal UI.

crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

basemind

Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

sluicer

Turn any web page, or a whole site, into structured data, with no model in the loop: every record a page declares with where each value came from, fields learnt from one example and checked on every page, crawls, feeds and archives.

crawlberg-hermes-plugin

High-performance web crawling engine with bindings for 11 languages

web-scout-ai

Web research that reads the sources—search, scrape HTML, JS sites and documents, then return cited answers with a complete audit trail.

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

basemind-hermes-plugin

Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

docpull

Local-first Python CLI and MCP server for versioned, cited context from web and document sources.

redis-queue-tool

Distributed task redisqueue(最简单python分布式函数调度框架)

gnosis-markdown

Web scraper & crawler → clean Markdown with byte-level SHA-256 provenance + WARC archival for LLM/RAG knowledge bases

crawl4ai-mcp-llm

Web crawling tool that integrates with AI assistants via the MCP

octocrawl-client

Octocrawl: readable Markdown and checked fields from public web pages, for AI agents, RAG and research. Failures come back with a reason. Self-host with REST, SDK or MCP.

healix

Crawl a website, classify and extract every page element, and generate self-healing Selenium/Playwright automation.