Datafusion Python Packages

ibis-framework

the portable Python dataframe library

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

polars-bio

Blazing-Fast Bioinformatic Operations on Python DataFrames

dbcrust

Fast psql-style database CLI/workbench for PostgreSQL, MySQL, SQLite, ClickHouse, MongoDB, Elasticsearch, and SQL over Parquet/CSV/JSON — with optional AI, SSH/Vault, Python & Django.

peql

A policy-enforcing SQL engine

ematix-flow

Move data between databases, files, and streams from Python.

datap-rs-client

An opinionated config-driven Rust server that publishes Parquet and Delta datasets as fast, typed HTTP APIs from local disk or object storage, with interchangeable DuckDB or Arrow+DataFusion backends, JSON and Arrow IPC output, and production-ready features like auth, metrics, and hot reloads.

micromegas

Unified observability for logs, metrics and traces — low overhead instrumentation (Rust, Unreal Engine 5), OTLP ingestion, SQL analytics on DataFusion over Parquet, presentation through notebooks, Grafana and Python. Self-hosted.

timeseries-table-format

Rust-native time-series table format with gap/overlap tracking and SQL queries

deltafunnel

Fast, lightweight Delta Lake to SQL Server loads without Spark or ODBC.

roapi

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

roapi-http

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

datafusion-extra-functions

Various additional function packages for Apache DataFusion (unofficial)

nail-parquet

Fast parquet command line tool with many functions, nailed it!

narwhals-datafusion

Apache DataFusion backend for Narwhals, registered through the narwhals.plugins entry point.

datap-rs

An opinionated config-driven Rust server that publishes Parquet and Delta datasets as fast, typed HTTP APIs from local disk or object storage, with interchangeable DuckDB or Arrow+DataFusion backends, JSON and Arrow IPC output, and production-ready features like auth, metrics, and hot reloads.

strake

The Data Layer for AI. A high-performance federated SQL engine that gives AI agents governed, zero-copy access to your entire data stack (Postgres, S3, APIs).

columnq-cli

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

irisdb

Ship the decoder with the data. Self-decoding datasets in Rust, with WebAssembly decoders that any engine can run.

bigtableql

SQL Query Layer for Google Cloud Bigtable

pdq

Fast secondary index for parquet files

beacon-datalake-cli

A high-performance climate data lake query engine supporting subsetting for zarr, netcdf, parquet, arrow ipc, csv, delta and iceberg

lakesoul

LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.

samkhya

Engine-agnostic Rust SDK for portable, feedback-driven cardinality correction in embedded analytical engines: DataFusion, DuckDB, Polars, Postgres, Iceberg, Arrow, GPU, Python. One Corrector trait — GBT, TabPFN-2.5, LLM. LpJoinBound never-regress clamp. Apache-2.0.