Distributed Python Packages

modal

SDK libraries for Modal

portalocker

An easy library for Python file locking. It works on Windows, Linux, BSD and Unix systems and can even perform distributed locking. Naturally it also supports the with statement.

lightgbm

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

optuna

A hyperparameter optimization framework

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

tensorflow

An Open Source Machine Learning Framework for Everyone

arq

Fast job queuing and RPC in python with asyncio and redis.

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

tensorflow-cpu

An Open Source Machine Learning Framework for Everyone

modin

Modin: Scale your Pandas workflows by changing a single line of code

svix-ksuid

A pure-Python KSUID implementation

pottery

Redis for humans. 🌎🌍🌏

tf-nightly

An Open Source Machine Learning Framework for Everyone

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

portal

Fast and reliable distributed systems in Python

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides a universal API for building parallel applications in the cloud ☁️🚀

kubeflow

Universal Python SDK to run AI workloads on Kubernetes

tensorflow-cpu-aws

An Open Source Machine Learning Framework for Everyone

tensorflow-intel

An Open Source Machine Learning Framework for Everyone

hazelcast-python-client

Hazelcast Python Client

vineyard

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

vineyard-bdist

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)