Spider Python Packages

spider-client

Python, Javascript, and Rust libraries for the Spider Cloud API.

368K 26 9

bilibili-api-python

哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api

301K 4K 619

crawlerdetect

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

163K 44 12

icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

77K 929 180

scrapeops-scrapy

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

8K 38 13

feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

7K 4K 544

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

6K 0 0

pyfreeproxy

FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理，支持爬取数十个免费代理分享源，支持自定义规则代理筛选，爬虫与数据分析必备，每日更新海量免费代理。)

6K 479 83

douyin-tiktok-scraper

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

5K 19K 3K

grab

Web Scraping Framework

5K 2K 276

jvav

[NSFW] Useful tools for crawling adult learning resources.

4K 65 10

decryptlogin

DecryptLogin: APIs for loginning some websites by using requests.

3K 3K 745

streamget

⚡ StreamGet - A Multi-Platform Live Stream Parser Library for Python

2K 235 38

papercrawlerutil

a collection of utils

2K 18 6

aio-scrapy

Implement scrapy with asyncio

2K 71 10

scrapy-cffi

An asyncio-style web scraping framework inspired by Scrapy, powered by curl_cffi, supported TLS, Http/Websocekt.

2K 8 0

bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

2K 1K 88

scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

2K 3K 582

photon

Incredibly fast crawler designed for OSINT.

2K 13K 2K

waifuboard

WaifuBoard is an asynchronous Python API module that can download images, image tags, and metadata from ACG image collection websites such as Danbooru, Safebooru, and Yandere. It can be easily integrated into any AI image model training process.

2K 2 0