Parallelization Python Packages

swifter

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

covalent

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

sdaxen-python-utilities

Useful tools for common Python tasks.

gamma-pytools

Foundational tools for BCG X's data science packages.

parmap

Easy to use map and starmap python equivalents

sciris

Fast, flexible tools to simplify scientific Python

bonobo

Extract Transform Load for Python 3.5+

dprs

Directed percolation and related models implemented in Rust and wrapped in Python

parajax

⚡ Automatic vectorization and parallelization of JAX-based functions

cova

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

parl

A high-performance distributed training framework for Reinforcement Learning

django-collectfaster

Parallel file copying for Django's collectstatic.

bluepyparallel

Provides an embarrassingly parallel tool with sql backend

aloisius

A Python library to create/update/delete AWS CloudFormation stacks in parallel

covalent-awslambda-plugin

Executor plugin interfacing Covalent with AWS Lambda

covalent-ssh-plugin

Executor plugin interfacing Covalent with remote backends using SSH

covalent-slurm-plugin

Executor plugin interfacing Covalent with Slurm

qc-parallelizer

A Python module for optimally combining and distributing quantum circuits

covalent-awsbatch-plugin

Executor plugin interfacing Covalent with AWS Batch

covalent-ecs-plugin

Executor plugin interfacing Covalent with Amazon ECS Fargate

covalent-braket-plugin

Executor plugin interfacing Covalent with Amazon Braket Hybrid Jobs

covalent-gcpbatch-plugin

Executor plugin interfacing Covalent with GCP Batch

sweepexp

A python package for running parallel experiments across parameter grids with MPI.

pystream-pipeline

A simple and fast real-time data pipeline constructor package for Python