Crawl Python Packages

tavily-python

The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.

xhs

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

nudecrawler

Crawl telegra.ph searching for nudes!

webclaw

Official Python SDK for the Webclaw web extraction API

sitecrawl

Simple Python module to crawl a website and extract URLs

crawlist

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

crawldiff

📜 git log for any website. Track changes with AI-powered summaries. Powered by Cloudflare /crawl.

renfepy

🚂 Train search CLI program and library written in Python

font-obfuscator

字体反爬、字体混淆工具是一个用于混淆字体文件的工具，可以将字体文件中的字形进行混淆，从而防止字体文件被直接提取出来。Font Obfuscator is an open-source Python library designed to prevent web scraping by obfuscating the character-to-glyph mapping in font files.

fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

preparser

a slight preparser to help parse webpage content or get request from urls,which supports win, mac and unix.

xhs-fork

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

universal-utils

python utils

spideyx

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.