Scrapyd Python Packages

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

10K 39 13

logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

4K 94 22

aio-scrapy

Implement scrapy with asyncio

2K 71 10

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

1K 3K 585

gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

1K 4K 649

spider-admin-pro

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看和爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

726 614 93

spiderkeeper

admin ui for scrapy/open source scrapinghub

285 3K 487

scrapyduler

Scrapyd module that schedules scrapy spiders by time.

131 1 0

scrapyd-heroku

Wrapper for running Scrapyd in Heroku or locally as a service

117 19 34

spidermanager

admin ui for scrapy/open source scrapinghub

111 2 1

scrapyq

Scrapyd queue management using Redis.

78 2 0

scrapyd-mongodb

Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management

64 17 8

scrapyd-egg-checksum

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

59 0 0

scrapyc

CLI and client library for scrapyd. Done right.

55 3 0

gerapy-team

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

53 4K 649

scrapyd-dash

Scrapyd Dashboard

46 8 1

scrapymon

Simple Web UI for Scrapy spider management via Scrapyd

32 50 10

scrapyd-ui

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

28 3K 585