Data Lake Python Packages

dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

lakefs-sdk

lakeFS - Data version control for your data lake | Git for data

lakefs

lakeFS - Data version control for your data lake | Git for data

lakefs-client

lakeFS - Data version control for your data lake | Git for data

dlt-cratedb

dlt destination adapter for CrateDB

aligned

The DBT of ML, as Aligned describes data dependencies in ML systems, and reduce technical data debt

nodestream

A Declarative framework for Building, Maintaining, and Analyzing Graph Data

dlt-core

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

s4-codec

GPU-accelerated transparent compression S3-compatible storage gateway. Drop-in replacement for AWS S3 endpoints; cuts your S3 bill 50-80% with no app changes (Rust, nvCOMP, zstd).

s4fs

GPU-accelerated transparent compression S3-compatible storage gateway. Drop-in replacement for AWS S3 endpoints; cuts your S3 bill 50-80% with no app changes (Rust, nvCOMP, zstd).

timeseries-table-format

Rust-native time-series table format with gap/overlap tracking and SQL queries

nodestream-plugin-semantic

A plugin for nodestream to ingest semantic data with embeddings

nodestream-plugin-dotenv

A plugin to nodestream for loading environment variables from a .env file

ztract

Extract mainframe EBCDIC data using COBOL copybooks. Zero MIPS. Pure Python + Cobrix engine.

lakexpress-mcp

MCP server for LakeXpress

nodestream-plugin-meta

A Meta Plugin for Nodestream that Lets you Build a Graph of Your Graph Schema

nodestream-plugin-pedantic

A nodestream plugin that provides a series of audits to ensure high quality and consistent nodestream projects.

etlx-wrapper

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

virtual-parquet

🗜️Compressing Parquet files using functions (TRL @NeurIPS'24, EDBT Best Demo'25)

lakefs-sdk-async

lakeFS API

dlt-dataops

dlt is an open-source python-first scalable data loading library that does not require any backend to run.

vulcan-sql

VulcanSQL turns your SQL templates into data APIs for efficient data sharing. No backend skills required. Empower your data sharing, faster.