Pyarrow Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

pyarrow-stubs

Type annotations for pyarrow

ibis-framework

the portable Python dataframe library

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

vgi-python

Python SDK for the VGI (Vector Gateway Interface) protocol — host DuckDB scalar, table, aggregate, and table-in-out functions in an external worker over Arrow IPC.

datagrunt

Datagrunt is a Python library designed to simplify the way you work with CSV, Excel, Parquet, and PDF files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.

daffy

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

timedatamodel

A lightweight Pythonic data model for time series data, interoperable with NumPy, Pandas and Polars.

jetxl

Blazing fast xlsx generator 🚀

keyedstablehash

Stable, keyed hashing for Python objects and columnar data. Think `stablehash`, but with SipHash-like keyed PRF semantics so hashes are deterministic for a given key and resistant to adversarial inputs.

pytest-ditto-pandas

Snapshot regression testing for DataFrames in pytest, with minimal ceremony, pluggable recorders, and local or remote storage backends.

pytest-ditto-pyarrow

Snapshot regression testing for DataFrames in pytest, with minimal ceremony, pluggable recorders, and local or remote storage backends.

libjay

Independent, modern implementations of the J and APL array languages: parallel and vectorized, embeddable from Rust, Python, and C

pytest-ditto-pickle

Snapshot regression testing for DataFrames in pytest, with minimal ceremony, pluggable recorders, and local or remote storage backends.

fast-xml-flattener

Rust-powered Python library for flattening nested XML into JSON, dict, CSV and Parquet. 4–7× faster than xmltodict.

schema2pyarrow

Converts AsyncApi and JsonSchema to PyArrow schema

fsspec-utils

fsspec utils

pytest-ditto-polars

Snapshot regression testing for DataFrames in pytest, with minimal ceremony, pluggable recorders, and local or remote storage backends.

finvizp

Async, Arrow-native Python client for the public Finviz data surface.

db2ixf

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

exness-data-preprocess

Professional forex tick data preprocessing with unified DuckDB storage, Phase7 OHLC schema, and sub-15ms query performance

pymetagen

Metadata Generator

sql2arrow

SQL2Arrow, short for 'SQL to Arrow,' is a Python library that provides convenient and high-performance methods to parse INSERT SQL statements into Arrow arrays. It is particularly useful for analyzing data dumped by mysqldump or other tools.

rxls

Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow