Data Streaming Python Packages

dagshub

DagsHub client libraries

streamdaq

🦆 Stream-first data quality monitoring in Python! Learn more: https://arxiv.org/abs/2506.06147

streamxl

Stream large Excel files with constant memory. 46x faster than openpyxl, handles multi-sheet workbooks. Rust-powered, Python API.

pystreamxl

Stream large Excel files with constant memory. 46x faster than openpyxl, handles multi-sheet workbooks. Rust-powered, Python API.

lorrystream

A lightweight and polyglot stream-processing library, to be used as a data backplane-, message relay-, or pipeline-subsystem.

dbt-flink-adapter

Adapter for dbt that executes dbt pipelines on Apache Flink

wax-ml

A Python library for machine-learning and feedback loops on streaming data

multiflow

A Python multithreading library for data processing pipelines, data streaming, etc.

fincore-py

Event-driven financial data and analytics framework for building real time market infrastructure.

shuflr-client

Stream shuffled records from huge JSONL/zstd corpora into PyTorch DataLoaders via Rust server, Python client, custom wire protocol with ~6× compression

affirmative-sampling

Reference implementation of the Affirmative Sampling algorithm by Jérémie Lumbroso and Conrado Martínez (2022). 🍀

randomhash

A simple, time-tested, family of random hash functions in Python, based on CRC32 and xxHash, affine transformations, and the Mersenne Twister. 🎲

sotastream

A library for data streaming and augmentation