Webcrawler Python Packages

decrypt-cookies

Get browser cookies and logins. Easily make a request using the authorization data from your browser.

gne

新闻网页正文通用抽取器 Beta 版.

scrapyrt

HTTP API for Scrapy spiders

markcrawl

Fast Python web crawler for AI & RAG ingestion — crawl, extract, and embed website content with one tool.

cloudfeaster

Cloudfeaster

onecrawler

An async python crawling framework for discovering URLs, extracting links, and scraping structured content.

scrapegraph-mcp

MCP server for ScapeGraph API integration

web-scraper-toolkit

AI-first web scraping engine with stealth bypass, MCP server, and multimodal output (Markdown, JSON, PDF) for agents and automation.

googlenewsscraper

Scrapes Google News article data

hydrosensorreader

Tools to read files from probes, sensors, or anything used in hydrogeology.

mseep-scrapegraph-mcp

ScapeGraph MCP Server

noizze-crawler

A web page crawler PyPI Package which returns (title, image, description)

sslproxies24

Obtenha e valide vários Proxys diariamente com este pacote. (Thread).

generalnewsextractor

新闻网页正文通用抽取器 Beta 版.

webpage-image-downloader

Python tool for finding and saving images from webpages.

catch

Web crawler with built in parsers usng latest Python technologies

scrapio

Aysncio web crawling framework

django-easy-scraper

Dango app to scrape web page

trawler

A data gathering framework to search and get information from web sources

korean-news-crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

wikilink

A multiprocessing web-scraping application to scrape wiki pages and find minimum number of links between two given wiki pages.

webscraping-tools

Tools to make webscraping easier

spydy

基于Pipeline的爬虫框架， 工作流非常简单、直观， 而且支持异步。light-weight high-level web-crawling framework

mindfactory-crawling

A crawler for mindfactory.de