File Processing Python Packages

netcdf-scm

Simple wrappers for processing netcdf files for use in simple climate models

papercrawlerutil

a collection of utils

pyrio

Functional-style Streams API

hsort

A high-performance sorting and merging engine optimized for processing large-scale data files, with stable multi-key sorting support for fixed-length, variable-length, and CSV formats.

bgate-unix

High-performance, hardware-aware Layer 0 deduplication engine. Uses 4-tier short-circuit logic (Size → Fringe → Full Hash) with xxHash128 to eliminate exact duplicates before they enter your processing pipeline. Python 3.11+, uv, SQLite.

alterx

A powerful file processing toolkit for batch transformations of HTML, JSON, TOML, XML, and YAML files

lwd-utils

rename-version of PaperCrawlerUtil

faster-os

6800% faster "os" module replacement. A drop-in replacement for Python's standard 'OS' module. Fully-rewritten, optimized, and speeded-up functions, that replace ones in the os.path module.

takin

A Python toolkit for file processing, text cleaning and data splitting. 文件处理，文本清洗和数据划分的python工具包。