Parallel Processing Python Packages

fastcore

Python supercharged for the fastai library

p-tqdm

Parallel processing with progress bars

accelerator

The Accelerator is a tool for fast and reproducible processing of eBay-scale datasets on a single computer.

envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

fastyaml-rs

Parse YAML at Rust speed. Full 1.2.2 spec, built-in linter, parallel processing. Native bindings for Python & Node.js.

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

glide

Easy ETL

dotflow

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

structai

StructAI offers a robust toolkit for LLM interaction—such as structured outputs, context management, and parallel execution.

smudgy

Python package for Point Cloud smoothing and interpolation with C++ and OpenMP backend.

grizzlars

High-performance DataFrame library written in C++ with Python bindings.

hypergol

Hypergol is a Data Science/Machine Learning productivity toolkit to accelerate any projects into production with autogenerated code, standardised structure for data and ML and parallel processing out-of-the-box.

exp

python experiment management toolset

octochains

A lightweight, zero-dependency Python framework for parallel, isolated, collaborative AI reasoning.

pyrallel-lib

Yet another easy-to-use python3 parallel library for humans.

parallelqueue

Parallel queueing models for SimPy

tfce-mediation

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE

preprocess-nlp

A fast framework for pre-processing (Cleaning text, Reduction of vocabulary, Feature extraction and Vectorization). Implemented with parallel processing using custom number of processes.

pyexpool

Python Multi-Process Execution Pool: concurrent asynchronous execution pool with custom resource constraints (memory, timeouts, affinity, CPU cores and caching), load balancing and profiling capabilities of the external apps on NUMA architecture

mpunittest

Multi processing for the Python unittest library

abopus

Audiobook to Opus converter — efficient batch conversion optimized for voice content

mpfd

A decorator that makes a function into a parallel processor using multithreading.

file-swirl

Now sort you files with just one command.

easy-runner

A lightweight tool for managing and executing multiple parallel experiments.