Scrapy Python Packages

scrapy-playwright

🎭 Playwright integration for Scrapy

itemadapter

Common interface for data container classes

random-user-agent

A package to get list of user agents based on filters such as operating system, software name etc..

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

scrapy-zyte-api

Zyte API integration for Scrapy

advertools

advertools - online marketing productivity and analysis tools

scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

scrapy-splash

Scrapy+Splash for JavaScript integration

scrapy-redis

Redis-based components for Scrapy.

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

scrapy-rotating-proxies

use multiple proxies with Scrapy

scrapy-crawlera

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

scrapy-stealth

Stealthy Crawling. Maximum Results. A pluggable anti-bot and stealth framework for Scrapy.

scrapingant-client

ScrapingAnt API client for Python.

city-scrapers-core

Core functionality for City Scrapers projects

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

scrapy-scrapingbee

JavaScript support and proxy rotation for Scrapy with ScrapingBee.

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

scrapy-s3pipeline

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

ayugespidertools

使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，解放开发者的双手。