Crawlers Python Packages

is-bot

Detect bots/crawlers/spiders via user-agent string

extracto-scraper

AI-powered web scraper. Give it a URL and tell it what data you want — it handles the rest.

scrapy-proxy-headers

Add custom proxy headers to HTTPS requests in Scrapy

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

wikilink

A multiprocessing web-scraping application to scrape wiki pages and find minimum number of links between two given wiki pages.

smoothcrawler-cluster

🌍🕷🔗🕷🔗🕷🕸 Building crawler cluster humanly depends on SmoothCrawler.

spyck

⛏ An extensible framework for data mining