Lakehouse Python Packages

starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

pydoris-custom

Apache Doris is an easy-to-use, high performance and unified analytics database.

pydoris

Apache Doris is an easy-to-use, high performance and unified analytics database.

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

ailake

AI-Lake Format: self-contained Parquet + HNSW index — unifying tabular data, embeddings, and vector search in a single Iceberg-compatible file

dbt-fabric-samdebruyn

Maintained and extended fork combining dbt-fabric and dbt-fabricspark

apache-gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

lakehouse-engine

The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.

sqllocks-spindle

Multi-domain, schema-aware synthetic data generator for Microsoft Fabric. 13 domains, billion-row scale, statistically calibrated. Lakehouse · Warehouse · SQL DB · Eventhouse writers.

lakehouse-plumber

The Metadata Driven framework for Databricks Lakeflow Declarative Pipelines (formerly Delta Live Tables). Metadata framework that generates production ready Pyspark code for Lakeflow Declarative Pipelines

dbt-doris

Apache Doris is an easy-to-use, high performance and unified analytics database.

lakebench

A multi-modal Python library for benchmarking lakehouse engines and ELT scenarios, supporting both industry-standard and novel benchmarks.

akad-framework

Lightweight Python framework for defining, enforcing, and monitoring data contracts on batch datasets

ytsaurus-spyt

YTsaurus is a scalable and fault-tolerant open-source big data platform.

databend

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

redpanda-polaris-catalog-python

Apache Polaris, the interoperable, open source catalog for Apache Iceberg

laketower

Oversee your lakehouse

timeseries-table-format

Rust-native time-series table format with gap/overlap tracking and SQL queries

apache-polaris

Apache Polaris, the interoperable, open source catalog for Apache Iceberg

doris-mcp-server

Apache Doris MCP Server

data-hub-gandalf

SCD (Types 0-6), upsert, surrogate key, checksum, and guarded Delta merge library for Databricks over Spark Connect — you shall not pass... bad data.

space-datasets

Unified storage framework for machine learning datasets

sas2databricks

LLM-assisted migration of SAS analytics, transformations, and reports to Databricks (PySpark, Spark SQL, DLT, Workflows).

ibm-watsonxdata-mcp-server

Model Context Protocol (MCP) server for IBM watsonx.data - enables AI assistants to query and explore lakehouse data Resources