Spider Python Packages

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

crawlerdetect

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

jvav

[NSFW] Useful tools for crawling adult learning resources.

pyfreeproxy

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

decryptlogin

DecryptLogin: APIs for loginning some websites by using requests.

dcard-spider

A spider on Dcard. Strong and speedy.

streamget

⚡ StreamGet - A Multi-Platform Live Stream Parser Library for Python

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

papercrawlerutil

一套工具组，包括访问链接， 获取元素，抽取文件等等 也有已经实现好通过scihub获取论文的小工具，还有对于pdf转doc，文本翻译,代理连接获取以及通过api获取代理链接， PDF文件合并，PDF文件截取某些页，CSV，xls文件处理等

grab

Web Scraping Framework

chromeremote

A tool for Chrome remote dev debugging.

bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

aio-scrapy

Implement scrapy with asyncio

douyin-tiktok-scraper

🚀 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for posts, profiles, comments and playlists. Self-healing identity pool, PostgreSQL archive, one docker compose up.

soliterm

Solitaire for your terminal. Klondike, Spider, FreeCell and 9 more games, played by GNOME AisleRiot's rules with keyboard or mouse, sharing AisleRiot's stats

attestql

Audit text-to-SQL gold and predictions on PostgreSQL or SQLite with typed replay evidence.

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

spider-rs

Spider ported to Python

scrapy-cffi

An asyncio-style web scraping framework inspired by Scrapy, powered by curl_cffi, supported TLS, Http/Websocekt.

gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

photon

Incredibly fast crawler designed for OSINT.