Distributed Python Packages

modal

SDK libraries for Modal

portalocker

An easy library for Python file locking. It works on Windows, Linux, BSD and Unix systems and can even perform distributed locking. Naturally it also supports the with statement.

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

lightgbm

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

tensorflow

An Open Source Machine Learning Framework for Everyone

optuna

A hyperparameter optimization framework

arq

Fast job queuing and RPC in python with asyncio and redis.

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

tensorflow-cpu

An Open Source Machine Learning Framework for Everyone

modin

Modin: Scale your Pandas workflows by changing a single line of code

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

pottery

Redis for humans. 🌎🌍🌏

tf-nightly

An Open Source Machine Learning Framework for Everyone

tensorflow-intel

An Open Source Machine Learning Framework for Everyone

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

tensorflow-cpu-aws

An Open Source Machine Learning Framework for Everyone

svix-ksuid

A pure-Python KSUID implementation

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

tensorflow-aarch64

An Open Source Machine Learning Framework for Everyone

kubeflow

Universal Python SDK to run AI workloads on Kubernetes

hazelcast-python-client

Hazelcast Python Client

tensorflow-macos

An Open Source Machine Learning Framework for Everyone

tf-nightly-cpu

An Open Source Machine Learning Framework for Everyone