Pyspark Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

ibis-framework

the portable Python dataframe library

datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

chispa

PySpark test helper methods with beautiful error messages

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

synapseml

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

spark-nlp

State of the Art Natural Language Processing

pyspark-test

Testing library for pyspark, inspired from pandas testing module but for pyspark, to help users write unit tests.

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

tdigest

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

copybook

Python copybook parser

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

pbspark

protobuf pyspark conversion

cuallee

Possibly the fastest DataFrame-agnostic quality check library in town.

pyspark-extension

A library that provides useful extensions to Apache Spark and PySpark.

h2o-pysparkling-3-1

Sparkling Water provides H2O functionality inside Spark cluster

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.