Crawler Python Packages

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

firecrawl-py

🔥 Search, scrape, and clean the web for AI agents.

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

newspaper3k

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

firecrawl

🔥 Search, scrape, and clean the web for AI agents.

scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

crawlee

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

random-user-agent

A package to get list of user agents based on filters such as operating system, software name etc..

rnet

An ergonomic Python HTTP Client with TLS fingerprint

crawlerdetect

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

scrapy-zyte-api

Zyte API integration for Scrapy

geckordp

A client implementation of Firefox DevTools over remote debug protocol in python

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

wreq

An ergonomic Python HTTP Client with TLS fingerprint

jmcomic

Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

scrapy-redis

Redis-based components for Scrapy.

lilbee

Terminal-first local search and AI chat over your documents, code, and crawled websites. Semantic + hybrid search, vision OCR, auto-built wiki, browsable GGUF model catalog. Works as CLI, TUI, MCP server, REST API, or Python library. Offline by default, no sidecar services.

html-table-extractor

extract data from html table

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

scrapy-crawlera

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy