Consumer Gpu Python Packages

experts4bit-qlora

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

soup-cli

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

loggetta

Plan and run Mixture-of-Experts fine-tuning on one GPU: check the machine, fit the model, and keep a report of each run.

moe-l2

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

miniverl

verl for a single consumer GPU. PPO, GRPO and on-policy distillation on NVIDIA GPUs.

pocketllm

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

ultracompress

Historical UltraCompress research and public CLI: pack structure checks and SHA-256 fingerprints.

turboquant-vllm

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

sentinel-lm

zero-dependency C++/CUDA DL framework focused on single consumer GPU's

e4b

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

experts4bit

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

e4b-qlora

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

experts-mxfp4

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

zorac

Self-hosted LLM chat client with streaming UI for vLLM servers. Run Mistral-24B locally on RTX 4090/3090. Privacy-focused ChatGPT alternative for homelab/gaming PCs. Python/Rich terminal UI.

expertsnbit

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.