Crawler Python Python Packages

is-crawler

Crawler detection from User-Agent strings in 50 ns. Issues and pull requests welcome!

mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

airspider

An Easy-to-use and Fast Python Spider Framework

crawlist

A universal solution for web crawling lists

quick-crawler

A toolkit for quickly performing crawler functions

sitecrawl

Simple Python module to crawl a website and extract URLs

tezzcrawler

Crawl any site, starting from the sitemap and convert entire website into Markdown, making it easy for the LLMs to learn

baikes

百度百科简易爬虫

sneakpeek-py

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

extracto-scraper

AI-powered web scraper. Extract structured data using plain English.

fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

weibo-api-sdk

对微博m站API进行封装，免登陆获取微博数据