Parallel Computing Python Packages

joblib

Computing with Python functions.

ctranslate2

Fast inference engine for Transformer models

swifter

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

mpi4py

Python bindings for MPI

pqdm

Comfortable parallel TQDM using concurrent.futures

pyopencl

OpenCL integration for Python, plus shiny features

pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

arm-mango

Parallel Hyperparameter Tuning in Python

adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions

schwimmbad

A common interface to processing pools.

bodo

High Performance Data Processing in Python

executorlib

Up-scale python functions for high-performance computing (HPC)

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

tensorpac

Phase-Amplitude Coupling under Python

optuna-distributed

Distributed hyperparameter optimization made easy

ultranest

Fit and compare complex models reliably and rapidly. Advanced nested sampling.

pydisort

Modern Python & C++ API for the radiative transfer solver DISORT. Parallelized with PyTorch tensors; compile-free with: pip install pydisort.

cuda-cccl

CUDA Core Compute Libraries

pympipool

Up-scale python functions for high-performance computing (HPC)

adaptive-scheduler

Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky, or dask-mpi. :tada:

imops

Efficient parallelizable algorithms for multidimensional arrays to speed up your data pipelines

hyperactive

A unified interface for optimization algorithms and experiments

pmmc

Mesh-based Monte Carlo (MMC)

mtkahypar

Mt-KaHyPar (Multi-Threaded Karlsruhe Hypergraph Partitioner) is a shared-memory multilevel graph and hypergraph partitioner equipped with parallel implementations of techniques used in the best sequential partitioning algorithms. Mt-KaHyPar can partition extremely large hypergraphs very fast and with high quality.