Web Scraper Python Packages

firecrawl-py

The API to search, scrape, and interact with the web at scale. 🔥

7.2M 144K 8K

firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

1.3M 144K 8K

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

797K 68K 7K

contextractor

Crawl any website and extract clean, boilerplate-free main-content text as Markdown, plain text, JSON, HTML, or raw original HTML — ready for LLMs, RAG, and vector databases. Built on rs-trafilatura + Crawlee/Playwright. Ships as a CLI, npm library, Python package, and Apify Actor.

10K 0 2

crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

10K 263 22

lightnovel-crawler

Generate and download e-books from online sources.

9K 2K 470

docpull

Convert the public web into AI-ready Markdown with a local Python CLI/SDK/MCP crawler.

8K 24 2

basketball-reference-web-scraper

NBA Stats API via Basketball Reference

7K 547 130

linktrace

Async web crawler with rate limiting, robots.txt support, and broken link tracking

7K 1 0

cbbpy

A Python-based web scraper for NCAA basketball.

5K 28 7

facebook-page-scraper

Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

3K 276 82

webanalyzer-security

WebAnalyzer is a versatile tool for comprehensive domain analysis. It provides insights into WHOIS data, DNS records, subdomains, SEO, web technologies, and security. Designed for developers and cybersecurity enthusiasts, it offers detailed reports and modular features for in-depth website audits.

2K 21 3

markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

1K 455 55

crawloop

Self-healing web crawler — compile your LLM scraper into free, deterministic code that auto-heals when a site's layout changes.

900 1 0

getdocs

Crawl any documentation site into clean, local markdown for coding agents and RAG — with source-first repo cloning.

866 0 0

gli99

Web scraper for gifcities.org

806 4 0

ollama-downloader

A library and Hugging Face model downloader for Ollama.

694 5 1

cbse-results-scraper

CBSE-Results-Scraper is a command-line Python application that scrapes and downloads student reports of CBSE (Central Board of Secondary Education) Class XII Results 2019 from http://cbseresults.nic.in.

639 2 3