Data Processing Python Packages

pandera

A light-weight, flexible, and expressive statistical data testing library

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

nvidia-nvimgcodec-cu12

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

nvidia-dali-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

bytewax

Python Stream Processing

pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

bonobo

Extract Transform Load for Python 3.5+

cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.

dolma

Data and tools for generating and inspecting OLMo pre-training data.

kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

untruncate-json

Python library to repair truncated json. Translated directly from the typescript original version

nvidia-nvimgcodec-cu13

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

itertable

⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.

pipe21

Simple functional pipes for python

libertem

Open pixelated STEM framework

glide

Easy ETL

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

nvidia-nvimgcodec-cu11

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

spectroview

SPECTROview : A Tool for Spectroscopic Data Processing and Visualization.

redis-message-queue

Lightweight Python message queuing with Redis and built-in publish-side deduplication.

tha-csv-runner

A Tabular Helper API library that reads and writes CSVs with progress tracking, header validation, and structured per-row errors.

text-dedup

All-in-one text de-duplication