Pyspark Python Packages

narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

chispa

PySpark test helper methods with beautiful error messages

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

ibis-framework

the portable Python dataframe library

synapseml

Simple and Distributed Machine Learning

hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

spark-nlp

State of the Art Natural Language Processing

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

tdigest

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

pyspark-test

Testing library for pyspark, inspired from pandas testing module but for pyspark, to help users write unit tests.

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

pbspark

protobuf pyspark conversion

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

cuallee

Possibly the fastest DataFrame-agnostic quality check library in town.

pyspark-extension

A library that provides useful extensions to Apache Spark and PySpark.

copybook

Python copybook parser

h2o-pysparkling-3-1

Sparkling Water provides H2O functionality inside Spark cluster

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.