Apache Spark Python Packages

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

synapseml

Simple and Distributed Machine Learning

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

lakefs-sdk

lakeFS - Data version control for your data lake | Git for data

lakefs

lakeFS - Data version control for your data lake | Git for data

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

lakefs-client

lakeFS - Data version control for your data lake | Git for data

sparkql

sparkql: Apache Spark SQL DataFrame schema management for sensible humans

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

fasttrackml

Experiment tracking server focused on speed and scalability

graphframes-latest

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

aqueduct-core

Intelligent, self-healing Spark pipelines. Declarative. Observable. Autonomous.

sparkdq

A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

pyspark-asyncactions

Asynchronous actions for PySpark

sparkhistogram

Includes notes on using Apache Spark, with drill down on Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark. Also tools for stress testing, measuring CPUs' performance, and I/O latency heat maps. Jupyter notebooks examples for using various DB systems.

dsgrid-toolkit

Python package for working with demand-side grid projects, datasets and queries

mcp-apache-spark-history-server

MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.