Data Processing Python Packages

pandera

A light-weight, flexible, and expressive statistical data testing library

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides a universal API for building parallel applications in the cloud ☁️🚀

nvidia-nvimgcodec-cu12

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

bytewax

Python Stream Processing

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

nvidia-dali-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

dolma

Data and tools for generating and inspecting OLMo pre-training data.

nvidia-nvimgcodec-cu13

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

itertable

⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.

haupt

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

dataexcept

A comprehensive Python library providing structured, hierarchical exception classes for data science, data engineering, and ML pipeline workflows. Features 40+ specialized exceptions, logging helpers, and production-ready error handling for robust data processing systems.

py-jsonl

Zero-dependency Python library for JSON Lines — read, write, compress, and stream in one line of code.

excelreader-native

High-performance, low-allocation Excel (.xlsx, .xlsb, .xls, .csv) reader and writer for .NET 10, with Native AOT support and C++/Rust/Python bindings.

cratedb-toolkit

CrateDB Toolkit, an SDK for CrateDB and CrateDB Cloud.

csvql-query

SQL queries for CSV files. The analytical CSV query engine for AI agents cli: csvql.

spectroview

SPECTROview: An Open-Source Application for Interactive Spectroscopic Data Processing, Fitting, and Visualization

nvidia-nvimgcodec-cu11

A nvImageCodec library of GPU- and CPU- accelerated codecs featuring a unified interface

nvidia-dali-nightly-cuda120

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

nvidia-dali-cuda110

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

untruncate-json

Python library to repair truncated json. Translated directly from the typescript original version

fpstreams

Lazy Python pipelines for iterables and records, with bounded async concurrency.

cdp-backend

Data storage utilities and processing pipelines used by CDP instances.