Web Spider Python Packages

wrapper-tls-requests

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

linktrace

Async web crawler with rate limiting, robots.txt support, and broken link tracking

spidy-web-crawler

Spidy is the simple, easy to use command line web crawler.

talospider

talospider - A simple,lightweight scraping micro-framework

smoothcrawler

🕷🕸🌍 Building crawler humanly as different roles with different components.

smoothcrawler-cluster

🌍🕷🔗🕷🔗🕷🕸 Building crawler cluster humanly depends on SmoothCrawler.

spidey-py

Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.