Crawler Python Python Packages

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

endpointscanner

Website endpoint recon tool and rate limit tester that can bypass simple captchas and WAFs.

mlscraper

Scrape HTML automatically with machine learning.

airspider

An Easy-to-use and Fast Python Spider Framework

sitecrawl

Simple Python module to crawl a website and extract URLs

quick-crawler

A toolkit for quickly performing crawler functions

extracto-scraper

AI-powered web scraper. Give it a URL and tell it what data you want — it handles the rest.

crawlist

A universal solution for web crawling lists

baikes

简单的百度百科爬虫

tezzcrawler

Crawl any site, starting from the sitemap and convert entire website into Markdown, making it easy for the LLMs to learn

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

weibo-api-sdk

一个免登陆获取新浪微博数据的Python库，简单易用