Crawling Framework Python Packages

ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

acrawler

🔍 A powerful web-crawling framework, based on aiohttp.

exoskeleton

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

wsltools

Web Scan Lazy Tools - Python Package

spideyx

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

spriterx

基于python协程池、用法灵活的高性能爬虫框架