Pyarrow Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

pyarrow-stubs

Type annotations for pyarrow

ibis-framework

the portable Python dataframe library

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

keyedstablehash

Stable, keyed hashing for Python objects and columnar data. Think `stablehash`, but with SipHash-like keyed PRF semantics so hashes are deterministic for a given key and resistant to adversarial inputs.

daffy

Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.

timedatamodel

A lightweight Pythonic data model for time series data, interoperable with NumPy, Pandas and Polars.

fast-xml-flattener

Rust-powered Python library for flattening nested XML into JSON, dict, CSV and Parquet. 4–7× faster than xmltodict.

datagrunt

Datagrunt is a Python library designed to simplify the way you work with CSV files. It provides a streamlined approach to reading, processing, and transforming your data into various formats, making data manipulation efficient and intuitive.

fsspec-utils

fsspec utils

schema2pyarrow

Converts AsyncApi and JsonSchema to PyArrow schema

db2ixf

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

pysummaries

Generate beautiful summary tables from pandas, polars or pyarrow dataframes

exness-data-preprocess

Professional forex tick data preprocessing with unified DuckDB storage, Phase7 OHLC schema, and sub-15ms query performance

pdf2dataset

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

pqviewer

View Apache Parquet Files In Your Terminal

dremio-arrow

Dremio Arrow Flight Client

pymetagen

Metadata Generator

sql2arrow

This is a Python library that provides convenient and high-performance methods to parse INSERT SQL statements into Arrow arrays.

swemaps

Maps of Sweden in GeoParquet

biosets

A bioinformatics extension of 🤗 Datasets library, built for ML applications on biological and omics data, offering easy integration of metadata and low-code data management tools.

turntable-spoonbill

the portable Python dataframe library

xdlake

A loose implementation of the deltalake protocol, written in Python on top of pyarrow, focused on extensibility, customizability, and distributed data.

rxls

Reading both XLSX and XLSB files, fast and memory-safe, into PyArrow.