Spark Streaming Python Packages

databricks-labs-dqx

Databricks framework to validate Data Quality of pySpark DataFrames and Tables

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

freeza-offset

Spark stream consumption commit in kafka consumer group

prophecy-spark-ai

High-performance AI/ML library for Spark to build and deploy your LLM applications in production.