Crawl Python Packages

tavily-python

The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.

xhs

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

air-web

A lightweight package for crawling the web with the minimalist of code.

webclaw

Official Python SDK for the Webclaw web extraction API

nudecrawler

Crawl telegra.ph searching for nudes!

crawlist

A universal solution for web crawling lists

sitecrawl

Simple Python module to crawl a website and extract URLs

crawldiff

git log for any website. Track what changed on any site with AI-powered summaries.

renfepy

Python library for crawl trains from renfe

fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

preparser

a slight preparser to help parse webpage content or get request from urls,which supports win, mac and unix.

font-obfuscator

字体反爬、字体混淆工具是一个用于混淆字体文件的工具，可以将字体文件中的字形进行混淆，从而防止字体文件被直接提取出来。Font Obfuscator is an open-source Python library designed to prevent web scraping by obfuscating the character-to-glyph mapping in font files.

xhs-fork

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

universal-utils

python utils

spideyx

SpideyX - A Web Reconnaissance Penetration Testing tool for Penetration Testers and Ethical Hackers