Parallelization Python Packages

swifter

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

gamma-pytools

Foundational tools for BCG X's data science packages.

sciris

Fast, flexible tools to simplify scientific Python

bonobo

Extract Transform Load for Python 3.5+

parmap

Easy to use map and starmap python equivalents

sdaxen-python-utilities

Useful tools for common Python tasks.

dprs

Directed percolation and related models implemented in Rust and wrapped in Python

covalent

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

cova

Covalent Workflow Tool

parl

A high-performance distributed training framework for Reinforcement Learning

bluepyparallel

Provides an embarrassingly parallel tool with sql backend.

django-collectfaster

Parallel file copying for Django's collectstatic.

covalent-awslambda-plugin

Executor plugin interfacing Covalent with AWS Lambda

covalent-ssh-plugin

Executor plugin interfacing Covalent with remote backends using SSH

covalent-awsbatch-plugin

Executor plugin interfacing Covalent with AWS Batch

aloisius

A Python library to create/update/delete AWS CloudFormation stacks in parallel

covalent-ecs-plugin

Executor plugin interfacing Covalent with Amazon ECS Fargate

covalent-braket-plugin

Executor plugin interfacing Covalent with Amazon Braket Hybrid Jobs

covalent-slurm-plugin

Executor plugin interfacing Covalent with Slurm

covalent-gcpbatch-plugin

Executor plugin interfacing Covalent with GCP Batch

qc-parallelizer

A Python module for optimally combining and distributing quantum circuits

ml-impute

A package for synthetic data generation for imputation using single and multiple imputation methods.

pystream-pipeline

A simple and fast real-time data pipeline constructor package for Python

sweepexp

A python package for running parallel experiments across parameter grids with MPI.