Crawlers Python Packages

is-bot

Detect bots/crawlers/spiders via user-agent string

scrapy-proxy-headers

Handle custom proxy headers when making HTTPS requests through proxies in scrapy

extracto-scraper

AI-powered web scraper. Give it a URL and tell it what data you want — it handles the rest.

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

smoothcrawler-cluster

🌍🕷🔗🕷🔗🕷🕸 Building crawler cluster humanly depends on SmoothCrawler.

wikilink

Scraping the wiki pages and find the minimum number of links between two wiki pages

spyck

⛏ An extensible framework for data mining