Dynamic Batching Python Packages

gpubudget

Vram budgeting and tail-latency simulation for three models sharing one triton gpu

dynabatch

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

online-dynamic-batching

Online dynamic batching for faster LLM/VLM fine-tuning

batch-inference

Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.