Parallel Processing Python Packages

fastcore

Python supercharged for the fastai library

p-tqdm

Parallel processing with progress bars

accelerator

The Accelerator is a tool for fast and reproducible processing of eBay-scale datasets on a single computer.

envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

fastyaml-rs

Parse YAML at Rust speed. Full 1.2.2 spec, built-in linter, parallel processing. Native bindings for Python & Node.js.

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

dotflow

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

glide

Easy ETL

structai

StructAI offers a robust toolkit for LLM interaction—such as structured outputs, context management, and parallel execution.

grizzlars

High-performance DataFrame library written in C++ with Python bindings.

octochains

A lightweight, zero-dependency Python framework for parallel, isolated, collaborative AI reasoning.

hypergol

Hypergol is a Data Science/Machine Learning productivity toolkit to accelerate any projects into production with autogenerated code, standardised structure for data and ML and parallel processing out-of-the-box.

parallelqueue

Parallel queueing models for SimPy

tfce-mediation

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE

pyrallel-lib

Yet another easy-to-use python3 parallel library for humans.

exp

python experiment management toolset

tmi-viewer

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE

pyexpool

Python Multi-Process Execution Pool: concurrent asynchronous execution pool with custom resource constraints (memory, timeouts, affinity, CPU cores and caching), load balancing and profiling capabilities of the external apps on NUMA architecture

multithread-parallel-processing

Library support parallel processing with multi-thread

abopus

Audiobook to Opus converter — efficient batch conversion optimized for voice content

mpfd

A decorator that makes a function into a parallel processor using multithreading.

mpunittest

Multi processing for the Python unittest library

preprocess-nlp

A fast framework for pre-processing (Cleaning text, Reduction of vocabulary, Feature extraction and Vectorization). Implemented with parallel processing using custom number of processes.

easy-runner

A lightweight tool for managing and executing multiple parallel experiments.