PyRank

Webcrawler Python Packages

Python packages with the GitHub topic webcrawler. Sorted by relevance, with stars and monthly downloads.
saying121
decrypt-cookies

Get browser cookies and logins. Easily make a request using the authorization data from your browser.

9K 10 2
AIMLPM
markcrawl

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

7K 2 0
umrashrf
catch

Web crawler with built in parsers using latest Python technologies

3K 0 1
GeneralNewsExtractor
gne

新闻网页正文通用抽取器 Beta 版.

2K 4K 541
scrapinghub
scrapyrt

HTTP API for Scrapy spiders

2K 880 162
simonsdave
cloudfeaster

Cloudfeaster

2K 3 0
sayedshaun
onecrawler

An async Python crawling framework for discovering URLs, extracting links, and scraping structured content.

2K 1 0
GeminidSystems
googlenewsscraper

A Python package that scrapes Google News article data while remaining undetected by Google. Our scraper can scrape page data up until the last page and never trigger a CAPTCHA (download stats: https://pepy.tech/project/GoogleNewsScraper)

881 11 5
cgq-qgc
hydrosensorreader

This project provides tools to read files from probes, sensors, or anything used in hydrogeology.

839 8 2
ScrapeGraphAI
scrapegraph-mcp

ScapeGraph MCP Server

629 72 22
imyourboyroy
web-scraper-toolkit

A powerful, standalone web scraping toolkit using Playwright and various parsers.

519 5 2
riquedev
sslproxies24

Captura e validação de Proxys (Python).

477 0 0
superjcd
spydy

基于Pipeline的爬虫框架， 工作流非常简单、直观， 而且支持异步。light-weight high-level web-crawling framework

394 2 0
YUChoe
noizze-crawler

A web page crawler PyPI Package which returns (title, image, description)

346 0 1
rrmerugu
trawler

A data gathering framework to search and get information from web sources

319 2 2
kingname
generalnewsextractor

新闻网页正文通用抽取器 Beta 版.

298 4K 541
ScrapeGraphAI
mseep-scrapegraph-mcp

ScapeGraph MCP Server

274 72 22
A-Bak
webpage-image-downloader

Python tool for finding and saving images from webpages.

267 0 0
EdmundMartin
scrapio

Asyncio web crawling framework. Work in progress.

253 19 4
tranlyvu
wikilink

Scraping the wiki pages and find the minimum number of links between two wiki pages

241 10 4
Indigo-Coder-github
korean-news-crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

215 1 0
Aravindha1234u
socialscraper

Social Scraper is a python tool meant for Detection of Child Predators/Cyber Harassers on Social Media

199 60 12
ScrapeGraphAI
iflow-mcp-scrapegraph-mcp

ScapeGraph MCP Server

170 72 22
Jack-Tilley
webscraping-tools

Tools to make webscraping easier

163 2 1