Distributed Python Packages

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

portalocker

An easy library for Python file locking. It works on Windows, Linux, BSD and Unix systems and can even perform distributed locking. Naturally it also supports the with statement.

modal

SDK libraries for Modal

tensorflow

An Open Source Machine Learning Framework for Everyone

optuna

A hyperparameter optimization framework

lightgbm

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

arq

Fast job queuing and RPC in python with asyncio and redis.

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

modin

Modin: Scale your Pandas workflows by changing a single line of code

tensorflow-cpu

An Open Source Machine Learning Framework for Everyone

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

tf-nightly

An Open Source Machine Learning Framework for Everyone

pottery

Redis for humans. 🌎🌍🌏

tensorflow-intel

An Open Source Machine Learning Framework for Everyone

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

kubeflow

Universal Python SDK to run AI workloads on Kubernetes

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

tensorflow-cpu-aws

An Open Source Machine Learning Framework for Everyone

svix-ksuid

A pure-Python KSUID implementation

tensorflow-aarch64

An Open Source Machine Learning Framework for Everyone

hazelcast-python-client

Hazelcast Python Client

tensorflow-macos

An Open Source Machine Learning Framework for Everyone

tf-nightly-cpu

An Open Source Machine Learning Framework for Everyone