Spark Python Packages

sqlglot

Python SQL Parser and Transpiler

pyspark

Apache Spark - A unified analytics engine for large-scale data processing

delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

sqlglotrs

Python SQL Parser and Transpiler

databricks-labs-dqx

Databricks framework to validate Data Quality of pySpark DataFrames and Tables

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

mleap

MLeap: Deploy ML Pipelines to Production

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

synapseml

Simple and Distributed Machine Learning

sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

visions

Type System for Data Analysis in Python

hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

delta-sharing

An open protocol for secure data sharing

koalas

Koalas: pandas API on Apache Spark

pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

spark-nlp

State of the Art Natural Language Processing

sqlglotc

Python SQL Parser and Transpiler

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

flytekit

Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.