Parquet Python Packages

pyarrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

influxdb3-python

Python module that provides a simple and convenient way to interact with InfluxDB 3.0.

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

parquet-tools

easy install parquet-tools

parquet-metadata

Dump metadata about a Parquet file.

awkward0

Manipulate arrays of complex data structures as easily as Numpy.

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

lonboard

Fast, interactive geospatial data visualization in Jupyter.

quilt3

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

chile-hub

Datos públicos de Chile curados, normalizados y validados: geografía, demografía, economía, salud, educación y distritos electorales. Listos para consumir en una línea de código con Polars, DuckDB, SQLite y Excel.

dataprof

Data profiling and quality gates for CSV, JSON, Parquet and DataFrames. Rust core, Python API.

litelink

An embedded Iceberg storage engine for append-only data

airflow-provider-xlsx

Airflow operators for converting XLSX files from/to Parquet/CSV/JSON

witan-sdk

Python client, CLI and local node for WITAN, the agent-to-agent knowledge market. pip install witan-sdk, or run the node as a container: ghcr.io/kor-jongwon/witan-node

infino

Embedded retrieval library built on Parquet. Fast, efficient, and scalable.

brain-mcp

The recorder for your AI conversations — local-first MCP memory that keeps byte-exact copies of your Claude Code, Codex and Pi sessions (Claude Code deletes its own after 30 days) and answers with a citation or abstains. DuckDB.

polars-config-meta

A Polars plugin for persistent DataFrame-level metadata

dbcrust

Fast psql-style database CLI/workbench for PostgreSQL, MySQL, SQLite, ClickHouse, MongoDB, Elasticsearch, and SQL over Parquet/CSV/JSON — with optional AI, SSH/Vault, Python & Django.

dask-deltatable

A Delta Lake reader for Dask

ml4t-data

Market data acquisition, storage, and update workflows for machine learning for trading.

cnequity

中国 A 股数据基础设施。55个日更数据集：股票行情、基本面、期货数据、资金面、公告事件、指数行业、宏观与风险。行级溯源、PIT 语义、复权与历史成分内置，MCP 原生。自托管，零注册、零 API Token

marimo-export

📦 Prepare selected marimo notebook states once, then consume verified exports from Python, TypeScript, agents, or custom applications

hflow

SDK for robotics teams to verify the quality of their data used for AI model training.