Apache Spark Python Packages

mlflow-skinny

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow-tracing

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

sparkmeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

synapseml

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

lakefs-sdk

lakeFS - Data version control for your data lake | Git for data

lakefs

lakeFS - Data version control for your data lake | Git for data

quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

pyspark-stubs

Apache (Py)Spark type annotations (stub files).

pysail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

pysparkling

A pure Python implementation of Apache Spark's RDD and DStream interfaces.

lakefs-client

lakeFS - Data version control for your data lake | Git for data

graphframes-latest

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

open-lakehouse-contract

Open Lakehouse Contract (OLC) is an open YAML specification for declaring a data product's schema, quality rules, transformations, lineage, ownership, service levels (SLAs and SLOs) and materialisation.

sparkql

sparkql: Apache Spark SQL DataFrame schema management for sensible humans

sparkdq

A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

convalesce-emit

Client libraries that forward a data tool's own output to Convalesce, unchanged.

odctl

A curated collection of open source technologies and the odctl CLI for running a local Docker-based data platform to experiment with modern data architecture and MLOps.

fasttrackml

Experiment tracking server focused on speed and scalability

kubeflow-spark-api

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

spylon

Utilities to work with Scala/Java code with py4j