Distributed Systems Python Packages

py4j

Py4J enables Python programs to dynamically access arbitrary Java objects

xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

typeid-python

Python implementation of TypeIDs: type-safe, K-sortable, and globally unique identifiers inspired by Stripe IDs

faststream

FastStream is a powerful and easy-to-use asynchronous Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS, MQTT and Redis.

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

faust-streaming

Python Stream Processing. A Faust fork

xgboost-cpu

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

pyro5

Pyro 5 - Python remote objects

torchft-nightly

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

safe-init

Safe Init is a Python library that enhances AWS Lambda functions with advanced error handling, logging, monitoring, and resilience features, providing comprehensive observability and reliability for serverless applications.

rucio-clients

Rucio - Scientific Data Management

eventsourcing

A library for event sourcing in Python.

sherlock

Easy distributed locks for Python with a choice of backends.

vineyard

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

pegasus-wms-api

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

pegasus-wms-common

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

streaq

Fast, async, fully-typed distributed task queue via Redis streams

pysyncobj

A library for replicating your python class between multiple servers, based on raft protocol

redis-dict

Python dictionary with Redis as backend, built for large datasets. Simplifies Redis operations for large-scale and distributed systems. Supports various data types, namespacing, pipelining, and expiration.

atomr-infer

Multi-runtime GPU + remote inference as a supervised actor system on atomr. OpenAI / Anthropic / Gemini / LiteLLM remote runtimes + vLLM / TensorRT / ORT / mistral.rs local; remote-only build compiles zero GPU deps.

google-vizier

Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

vineyard-bdist

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)