PyRank

Spark Python Packages

Python packages with the GitHub topic spark. Sorted by relevance, with stars and monthly downloads.
tobymao
sqlglot

Python SQL Parser and Transpiler

73.3M 9K 1K
apache
pyspark

Apache Spark - A unified analytics engine for large-scale data processing

49M 44K 29K
delta-io
delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

36M 9K 2K
databrickslabs
databricks-labs-dqx

Databricks framework to validate Data Quality of pySpark DataFrames and Tables

6.8M 430 124
tobymao
sqlglotrs

Python SQL Parser and Transpiler

4.7M 9K 1K
capitalone
datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

3.6M 651 161
graphframes
graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

2.9M 1K 267
apache
pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

2.3M 44K 29K
fugue-project
fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

2.3M 2K 101
combust
mleap

MLeap: Deploy ML Pipelines to Production

2.2M 2K 318
lucacanali
sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

2.2M 827 160
Microsoft
synapseml

Simple and Distributed Machine Learning

1.9M 5K 861
tobymao
sqlglotc

Python SQL Parser and Transpiler

1.7M 9K 1K
jupyter-incubator
hdijupyterutils

Jupyter magics and kernels for working with remote Spark clusters

1.5M 1K 445
jupyter-incubator
autovizwidget

Jupyter magics and kernels for working with remote Spark clusters

1.5M 1K 445
dylan-profiler
visions

Type System for Data Analysis in Python

1.5M 222 20
delta-io
delta-sharing

An open protocol for secure data sharing

1.4M 951 230
JohnSnowLabs
spark-nlp

State of the Art Natural Language Processing

1.2M 4K 742
databricks
koalas

Koalas: pandas API on Apache Spark

1.2M 3K 369
moj-analytical-services
splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

1M 2K 243
graphframes
graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

1M 1K 267
microsoft
dbt-fabricspark

A dbt adapter for Fabric Lakehouse backed by Apache Spark.

921K 63 36
fugue-project
fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

920K 2K 101
jelmerk
pyspark-hnsw

Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs

737K 302 59