Parallel Processing Python Packages

fastcore

Python supercharged for the fastai library

p-tqdm

Parallel processing with progress bars

envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.

accelerator

The Accelerator is a tool for fast and reproducible processing of eBay-scale datasets on a single computer.

pygmo

A Python platform to perform parallel computations of optimisation tasks (global and local) via the asynchronous generalized island model.

fastyaml-rs

Parse YAML at Rust speed. Full 1.2.2 spec, built-in linter, parallel processing. Native bindings for Python & Node.js.

rypipe

Format- and source-agnostic ingestion framework: a common execution runtime for turning record-oriented data sources into typed columnar data. Rust core, Python bindings; adapters live in separate packages.

async-batch-llm

Provider-agnostic framework for high-throughput LLM processing with async workers, automatic retries, rate limiting, and intelligent validation recovery.

dotflow

🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines

mordant

A blazing-fast Markdown parser and linter for Python, written in Rust.

smudgy

Python package for Point Cloud smoothing and interpolation with C++ and OpenMP backend.

mmap-chunker-core

Record-aligned byte-range planning for large immutable files, with zero-copy mmap framing, CLI, and a stable C ABI.

structai

StructAI offers a robust toolkit for LLM interaction—such as structured outputs, context management, and parallel execution.

grizzlars

High-performance DataFrame library written in C++ with Python bindings.

glide

Easy ETL

tfce-mediation

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE

octochains

A lightweight Python framework for parallel, isolated, collaborative AI reasoning.

hypergol

Hypergol is a Data Science/Machine Learning productivity toolkit to accelerate any projects into production with autogenerated code, standardised structure for data and ML and parallel processing out-of-the-box.

exp

python experiment management toolset

pyrallel-lib

Yet another easy-to-use python3 parallel library for humans.

parallelqueue

A package for simulating parallel queueing systems in Python.

pyexpool

Python Multi-Process Execution Pool: concurrent asynchronous execution pool with custom resource constraints (memory, timeouts, affinity, CPU cores and caching), load balancing and profiling capabilities of the external apps on NUMA architecture

mpfd

A decorator that makes a function into a parallel processor using multithreading.

tmi-viewer

Fast regression and mediation analysis of vertex or voxel MRI data with TFCE