Apache Spark Python Packages

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

synapseml

Simple and Distributed Machine Learning

sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

lakefs-sdk

lakeFS - Data version control for your data lake | Git for data

lakefs

lakeFS - Data version control for your data lake | Git for data

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

lakefs-client

lakeFS - Data version control for your data lake | Git for data

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

sparkql

sparkql: Apache Spark SQL DataFrame schema management for sensible humans

fasttrackml

Experiment tracking server focused on speed and scalability

graphframes-latest

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pysparkgateway

Connect to remote Spark clusters seamlessly.

sparkhistogram

Includes notes on using Apache Spark, with drill down on Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark. Also tools for stress testing, measuring CPUs' performance, and I/O latency heat maps. Jupyter notebooks examples for using various DB systems.

mcp-apache-spark-history-server

MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

pyspark-asyncactions

Asynchronous actions for PySpark

spylon

Utilities to work with Scala/Java code with py4j

dsgrid-toolkit

Python package for working with demand-side grid projects, datasets and queries