Crawler Python Python Packages

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

endpointscanner

Website endpoint recon tool and rate limit tester that can bypass simple captchas and WAFs.

airspider

A Fast and Light Python Spider Framework 🕷️

quick-crawler

A toolkit for quickly performing crawler functions

sitecrawl

Simple Python module to crawl a website and extract URLs

crawlist

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

extracto-scraper

AI-powered web scraper. Give it a URL and tell it what data you want — it handles the rest.

tezzcrawler

Crawl any site, starting from the sitemap and convert entire website into Markdown, making it easy for the LLMs to learn

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

baikes

简单的百度百科爬虫

fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

weibo-api-sdk

对微博m站API进行封装，免登陆获取微博数据