Spark Streaming Python Packages

databricks-labs-dqx

Databricks framework to validate Data Quality of pySpark DataFrames and Tables

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

freeza-offset

Spark stream consumption commit in kafka consumer group

prophecy-spark-ai

Toolbox for building Generative AI applications on top of Apache Spark.

polarway

Railway-oriented distributed reimplementation of Polars's fast Query Engine : hybrid storage in a Lakehouse & low-latency data streaming at scale. No JVM, forget executor memory / cores management... Function composition, improved Functors and Monadic-type usage for safe handling of your data as stream or from cold storage (any parquet-compatible)