Spider Python Packages

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

bilibili-api-python

哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api

crawlerdetect

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

jvav

[NSFW] Useful tools for crawling adult learning resources.

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

pyfreeproxy

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

grab

Web Scraping Framework

spider-rs

Spider ported to Python

papercrawlerutil

a collection of utils

decryptlogin

DecryptLogin: APIs for loginning some websites by using requests.

douyin-tiktok-scraper

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

waifuboard

Asynchronous API for downloading images, tags, and metadata from image board sites (e.g., Danbooru, Safebooru, Yandere). Ignore the downloaded files.

streamget

⚡ StreamGet - A Multi-Platform Live Stream Parser Library for Python

aio-scrapy

A high-level Web Crawling and Web Scraping framework based on Asyncio

bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

youdaodict

有道词典网页爬虫

cloudfeaster

Cloudfeaster

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

photon

Incredibly fast crawler designed for OSINT.

spkcspider

Your decentral spider for your digital identity

ant-nest

A simple and clear Web Crawler framework build on python3.6+