Spiders Python Packages

spidermon

Scrapy Extension for monitoring spiders execution.

scrapy-proxy-headers

Handle custom proxy headers when making HTTPS requests through proxies in scrapy

crawlerutils

Crawler Utils examples

cumtbscore

Student score interface for CUMTB

shub-cli

A CLI for dealing with the features of ScrapingHub

scrapy-dynamic-spiders

Dynamically generate spider subclasses based on existing spiders, including editing crawl rules and per-spider custom settings. Run crawls sequentially using crochet. Do both at the same time.