Vram Optimization Python Packages

moe-l2

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

sentinel-lm

zero-dependency C++/CUDA DL framework focused on single consumer GPU's

dynabatch

PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.

hypermcts

🚀 Autonomous Test-Time Compute Scaling & Dynamic KV-Cache Sparsification for Edge & Local LLMs | Pure PyTorch / Ollama MCTS Reasoning Engine (-65% VRAM)

nearlossless-context

Near-lossless long-context KV compression under fixed VRAM, critical-span retention, query-unknown discovery, and measured peak/quality Pareto.

gemma4-adaptive-router

Adaptive dual-tier serving for Gemma 4 on consumer 16GB GPUs. Complexity + real-time VRAM routing between vLLM E4B and llama.cpp 27B. Production stack with OpenWebUI, monitoring, and more.

ironsilo

Private, autonomous AI lab — turn your PC into a local-first intelligence platform without melting your GPU. Python + FastAPI + local LLMs.

sparsemma

INT8 Sparse Tensor Core GEMM for PyTorch — built for Windows