Distributed Systems Python Packages

py4j

Py4J enables Python programs to dynamically access arbitrary Java objects

xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

faststream

FastStream is an asynchronous Python framework for building event-driven applications. It brings together message broker integration, dependency injection, validation, testing utilities, and AsyncAPI documentation generation in a single toolkit

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

xgboost-cpu

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

typeid-python

Python implementation of TypeIDs: type-safe, K-sortable, and globally unique identifiers inspired by Stripe IDs

pyro5

Pyro 5 - Python remote objects

faust-streaming

Python Stream Processing. A Faust fork

safe-init

Safe Init is a Python library that enhances AWS Lambda functions with advanced error handling, logging, monitoring, and resilience features, providing comprehensive observability and reliability for serverless applications.

torchft-nightly

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

eventsourcing

A library for event sourcing in Python.

rucio-clients

Rucio - Scientific Data Management

pysyncobj

A library for replicating your python class between multiple servers, based on raft protocol

sherlock

Easy distributed locks for Python with a choice of backends.

puppetmaster-ai

Provider-neutral control plane for durable-state agent swarms: subprocess workers, leases, artifacts, memory, and deterministic stitching.

streaq

Fast, async, fully-typed distributed task queue via Redis streams

vineyard

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

pegasus-wms-api

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

pegasus-wms-common

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

vineyard-bdist

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

redis-dict

Python dictionary with Redis as backend, built for large datasets. Simplifies Redis operations for large-scale and distributed systems. Supports various data types, namespacing, pipelining, and expiration.

cachekit

Production-ready caching for Python with intelligent reliability features and Rust-powered performance