Crawlers Python Packages

is-bot

Detect bots/crawlers/spiders via user-agent string

scrapy-proxy-headers

Add custom proxy headers to HTTPS requests in Scrapy

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

extracto-scraper

AI-powered web scraper. Extract structured data using plain English.

smoothcrawler-cluster

🌍🕷🔗🕷🔗🕷🕸 Building crawler cluster humanly depends on SmoothCrawler.

wikilink

Scraping the wiki pages and find the minimum number of links between two wiki pages

spyck

Crawler framework