Quantization Python Packages

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

compressed-tensors

A safetensors extension to efficiently store sparse quantized tensors on disk

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

torchao

PyTorch native quantization for training and inference

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

auto-round

A simple and effective post training quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的后训练量化工具包

optimum-quanto

A pytorch quantization backend for optimum

sageattention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

onnx2tf

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

navec

Compact high quality word embeddings for Russian language

tensorflow-model-optimization

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

veloxquant-mlx

Fast KV-cache quantization for Apple Silicon (MLX) — 43 research-adapted compression methods with Metal kernels

turbovec

A vector index built on TurboQuant, written in Rust with Python bindings

gptqmodel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

llmcompressor

State-of-the-art LLM compression, built for production inference with vLLM

fastt5

⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.

tigris-ml

Ahead-of-time compiler that fits ML models on embedded devices with hard memory budgets.

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

discrete-key-value-bottleneck-pytorch

Implementation of Discrete Key / Value Bottleneck, in Pytorch

qonnx

QONNX: Arbitrary-Precision Quantized Neural Networks in ONNX

auto-gptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform