File Processing Python Packages

pyrio

Functional-style Streams API

papercrawlerutil

一套工具组，包括访问链接， 获取元素，抽取文件等等 也有已经实现好通过scihub获取论文的小工具，还有对于pdf转doc，文本翻译,代理连接获取以及通过api获取代理链接， PDF文件合并，PDF文件截取某些页，CSV，xls文件处理等

netcdf-scm

Simple wrappers for processing netcdf files for use in simple climate models

hsort

High-performance sorting and merging engine for large data files. Python extension with CLI and API support.

bgate-unix

High-performance, hardware-aware Layer 0 deduplication engine. Uses 4-tier short-circuit logic (Size → Fringe → Full Hash) with xxHash128 to eliminate exact duplicates before they enter your processing pipeline. Python 3.11+, uv, SQLite.

alterx

A powerful file processing toolkit for batch transformations of HTML, JSON, TOML, XML, and YAML files

lwd-utils

一套工具组，包括访问链接， 获取元素，抽取文件等等 也有已经实现好通过scihub获取论文的小工具，还有对于pdf转doc，文本翻译,代理连接获取以及通过api获取代理链接， PDF文件合并，PDF文件截取某些页，CSV，xls文件处理等

faster-os

6800% faster "os" module replacement. A drop-in replacement for Python's standard 'OS' module. Fully-rewritten, optimized, and speeded-up functions, that replace ones in the os.path module.

takin

A Python toolkit for file processing, text cleaning and data splitting. 文件处理，文本清洗和数据划分的python工具包。

triagehash

High-performance, hardware-aware Layer 0 deduplication engine. Uses 4-tier short-circuit logic (Size → Fringe → Full Hash) with xxHash128 to eliminate exact duplicates before they enter your processing pipeline. Python 3.11+, uv, SQLite.