Spark Python Packages

sqlglot

Python SQL Parser and Transpiler

pyspark

Apache Spark - A unified analytics engine for large-scale data processing

delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

databricks-labs-dqx

Databricks framework to validate Data Quality of pySpark DataFrames and Tables

sqlglotrs

Python SQL Parser and Transpiler

datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

sqlglotc

Python SQL Parser and Transpiler

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

mleap

MLeap: Deploy ML Pipelines to Production

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

synapseml

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

delta-sharing

An open protocol for secure data sharing

hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

visions

Type System for Data Analysis in Python

spark-nlp

State of the Art Natural Language Processing

koalas

Koalas: pandas API on Apache Spark

flytekit

Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.

pyspark-test

Testing library for pyspark, inspired from pandas testing module but for pyspark, to help users write unit tests.

pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs