Data Processing Python Packages

pandera

A light-weight, flexible, and expressive statistical data testing library

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

nvidia-nvimgcodec-cu12

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

bytewax

Python Stream Processing

kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

runhouse

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

nvidia-dali-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

dolma

Data and tools for generating and inspecting OLMo pre-training data.

bonobo

Extract Transform Load for Python 3.5+

untruncate-json

Python library to repair truncated json. Translated directly from the typescript original version

cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.

itertable

⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.

pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

nvidia-nvimgcodec-cu11

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

nvidia-nvimgcodec-cu13

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

pipe21

Simple functional pipes for python

libertem

Open pixelated STEM framework

redis-message-queue

Reliable Python message queuing with Redis and built-in deduplication. Publish once, process once, recover from crashes - across any number of producers and consumers.

py-data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

glide

Easy ETL

nvidia-dali-cuda110

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.