Scrapy Extension Python Packages

ayugespidertools

使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，解放开发者的双手。

scrapy-wayback-machine

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

scrapy-fieldstats

A Scrapy extension to log items coverage when the spider shuts down

scrapy-ua-rotator

Flexible and modern User-Agent rotator middleware for Scrapy, supporting Faker, fake-useragent, and custom providers.

scrapy-spiderstats-extension

将 Spider Stats 存储到 MongoDB 的扩展，可以用于爬虫监控和统计

scrapy-slackbot

A Scrapy extension for sending notification to Slack channels

scrapy-flaresolverr

Scrapy downloader middleware for FlareSolverr with multiple backends, concurrency control, and optional sessions and proxies.

scrapy-folder-tree

[MIRROR OF https://codeberg.org/sp1thas/scrapy-folder-tree] A scrapy pipeline which stores files using folder trees.