Quantization Python Packages

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

torchao

PyTorch native quantization and sparsity for training and inference

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

onnx2tf

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

llmcompressor

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

optimum-quanto

A pytorch quantization backend for optimum

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

sageattention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

tensorflow-model-optimization

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

auto-round

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

auto-gptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

navec

Compact high quality word embeddings for Russian language

gptqmodel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

aimet-torch

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

qonnx

QONNX: Arbitrary-Precision Quantized Neural Networks in ONNX

brevitas

Brevitas: neural network quantization in PyTorch

tqdb

Embedded vector database using the TurboQuant algorithm (arXiv:2504.19874) — zero training, 2-4 bit compression, fast inner-product search

fms-model-optimizer

FMS Model Optimizer is a framework for developing reduced precision neural network models.

aimet-onnx

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.