Distributed Systems Python Packages

xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

xgboost-cpu

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

faststream

Asynchronous Python framework for event-driven services. A thin client for Kafka, RabbitMQ, NATS, Redis and MQTT with full access to native broker features, plus AsyncAPI docs, in-memory tests and observability out of the box.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

typeid-python

Python implementation of TypeIDs: type-safe, K-sortable, and globally unique identifiers inspired by Stripe IDs

faust-streaming

Python Stream Processing. A Faust fork

fugue-sql-antlr

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

pyro5

Pyro 5 - Python remote objects

torchft-nightly

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

safe-init

Safe Init is a Python library that enhances AWS Lambda functions with advanced error handling, logging, monitoring, and resilience features, providing comprehensive observability and reliability for serverless applications.

xgboost-cu12

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eventsourcing

A library for event sourcing in Python.

google-vizier

Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

vineyard

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

vineyard-bdist

vineyard (v6d): an in-memory immutable data manager. (Project under CNCF, TAG-Storage)

pyresilience

Unified resilience patterns for Python — retry, circuit breaker, timeout, fallback, bulkhead, rate limiter, and cache in one decorator. Python's Resilience4j.

tapio-py

A Pekko-inspired actor toolkit for Python: typed, asyncio-native actors with supervision, Pydantic messages, and point-to-point remoting.

rucio-clients

Rucio - Scientific Data Management

streaq

Fast, async, fully-typed distributed task queue via Redis streams

pysyncobj

A library for replicating your python class between multiple servers, based on raft protocol

pegasus-wms-api

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

pegasus-wms-common

Pegasus Workflow Management System - Automate, recover, and debug scientific computations.

puppetmaster-ai

Provider-neutral control plane for durable-state agent swarms: subprocess workers, leases, artifacts, memory, and deterministic stitching.