PyRank

Webcrawler Python Packages

Python packages with the GitHub topic webcrawler. Sorted by monthly downloads, with stars and forks.
saying121
decrypt-cookies

Get browser cookies and logins. Easily make a request using the authorization data from your browser.

10K 14 3
scrapinghub
scrapyrt

HTTP API for Scrapy spiders

2K 884 160
GeneralNewsExtractor
gne

新闻网页正文通用抽取器 Beta 版.

1K 4K 537
simonsdave
cloudfeaster

Cloudfeaster Spider Development

1K 3 0
ScrapeGraphAI
scrapegraph-mcp

ScapeGraph MCP Server

730 115 38
GeminidSystems
googlenewsscraper

A Python package that scrapes Google News article data while remaining undetected by Google. Our scraper can scrape page data up until the last page and never trigger a CAPTCHA (download stats: https://pepy.tech/project/GoogleNewsScraper)

555 12 5
cgq-qgc
hydrosensorreader

This project provides tools to read files from probes, sensors, or anything used in hydrogeology.

552 8 2
AIMLPM
markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

535 5 1
imyourboyroy
web-scraper-toolkit

AI-first web scraping engine with stealth bypass, MCP server, and multimodal output (Markdown, JSON, PDF) for agents and automation.

378 6 2
sayedshaun
onecrawler

Async Python crawling library with sitemap discovery, browser-based extraction, and LLM-powered structured output — one Settings object drives it all.

332 1 0
ScrapeGraphAI
mseep-scrapegraph-mcp

ScapeGraph MCP Server

323 115 38
umrashrf
catch

Web crawler with built in parsers using latest Python technologies

302 0 1
riquedev
sslproxies24

Captura e validação de Proxys (Python).

251 0 0
ScrapeGraphAI
iflow-mcp-scrapegraph-mcp

ScapeGraph MCP Server

233 115 38
superjcd
spydy

基于Pipeline的爬虫框架， 工作流非常简单、直观， 而且支持异步。light-weight high-level web-crawling framework

209 2 0
YUChoe
noizze-crawler

A web page crawler PyPI Package which returns (title, image, description)

198 0 1
EdmundMartin
scrapio

Asyncio web crawling framework. Work in progress.

182 19 4
kingname
generalnewsextractor

新闻网页正文通用抽取器 Beta 版.

142 4K 538
Indigo-Coder-github
korean-news-crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

137 1 0
rrmerugu
trawler

A data gathering/trawling framework to search and get information from web sources like bing

130 2 2
scrapinghub
scrapyrt-with-params

HTTP API for Scrapy spiders

107 884 160
Aravindha1234u
socialscraper

Social Scraper is a python tool meant for Detection of Child Predators/Cyber Harassers on Social Media

99 62 12
albert-marrero
yugioh-scraper

Yu-Gi-Oh! Scraper is a project that crawls websites and APIs and extracts Yu-Gi-Oh! related data from their pages.

97 1 0
YUChoe
httpserver-mock

A web page crawler PyPI Package which returns (title, image, description)

85 0 1