Scrapy Python Packages

itemadapter

Common interface for data container classes

2.6M 69 13

scrapy-playwright

🎭 Playwright integration for Scrapy

1.2M 1K 160

random-user-agent

A package to get list of user agents based on filters such as operating system, software name etc..

397K 103 12

scrapfly-sdk

Official Python SDK for the Scrapfly platform: web scraping, screenshots, AI extraction, crawling, and a remote anti-bot browser. Integrates with Scrapy, LlamaIndex, and LangChain.

256K 55 15

advertools

advertools - online marketing productivity and analysis tools

168K 1K 242

scrapy-zyte-api

Zyte API integration for Scrapy

106K 41 22

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

79K 924 179

scrapy-splash

Scrapy+Splash for JavaScript integration

78K 3K 456

scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.

39K 233 27

scrapy-redis

Redis-based components for Scrapy.

39K 6K 2K

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

27K 952 352

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

26K 688 94

scrapy-zyte-smartproxy

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

25K 365 91

scrapy-crawlera

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

23K 365 91

scrapy-rotating-proxies

use multiple proxies with Scrapy

15K 774 158

scrapingant-client

ScrapingAnt API client for Python.

10K 43 5

city-scrapers-core

Core functionality for City Scrapers projects

9K 8 10

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

8K 4K 543