Parallel Computing Python Packages

joblib

Computing with Python functions.

swifter

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

ctranslate2

Fast inference engine for Transformer models

mpi4py

Python bindings for MPI

pqdm

Comfortable parallel TQDM using concurrent.futures

pyopencl

OpenCL integration for Python, plus shiny features

ultranest

Fit and compare complex models reliably and rapidly. Advanced nested sampling.

bodo

High Performance Scalable Data Processing in Python and SQL

arm-mango

Parallel Hyperparameter Tuning in Python

adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

schwimmbad

A common interface to processing pools.

mtkahypar

Mt-KaHyPar (Multi-Threaded Karlsruhe Hypergraph Partitioner) is a shared-memory multilevel graph and hypergraph partitioner equipped with parallel implementations of techniques used in the best sequential partitioning algorithms. Mt-KaHyPar can partition extremely large hypergraphs very fast and with high quality.

pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

cuda-cccl

CUDA Core Compute Libraries

imops

Efficient parallelizable algorithms for multidimensional arrays to speed up your data pipelines

adaptive-scheduler

Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky, or dask-mpi. :tada:

pydisort

Modern Python & C++ API for the radiative transfer solver DISORT. Parallelized with PyTorch tensors; compile-free with: pip install pydisort.

scitex-parallel

Thread/process pool parallel-execution utilities

tensorpac

Phase-Amplitude Coupling under Python

executorlib

Up-scale python functions for high-performance computing (HPC)

pmmc

Mesh-based Monte Carlo (MMC)

pyrxmesh

GPU-accelerated triangle mesh processing

optimagic

optimagic is a Python package for numerical optimization. It is a unified interface to optimizers from SciPy, NlOpt and other packages. optimagic's minimize function works just like SciPy's, so you don't have to adjust your code. You simply get more optimizers for free. On top you get diagnostic tools, parallel numerical derivatives and more.