Quantization Python Packages

ctranslate2

Fast inference engine for Transformer models

faster-whisper

Faster Whisper transcription with CTranslate2

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

torchao

PyTorch native quantization and sparsity for training and inference

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

onnx2tf

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

optimum-quanto

A pytorch quantization backend for optimum

llmcompressor

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

sageattention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

tensorflow-model-optimization

A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.

auto-round

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

navec

Compact high quality word embeddings for Russian language

auto-gptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

mblt-model-zoo

Mobilint Model Zoo Project

qonnx

QONNX: Arbitrary-Precision Quantized Neural Networks in ONNX

turbovec

A vector index built on TurboQuant, written in Rust with Python bindings

intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

aimet-torch

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

gptqmodel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

discrete-key-value-bottleneck-pytorch

Implementation of Discrete Key / Value Bottleneck, in Pytorch

fms-model-optimizer

FMS Model Optimizer is a framework for developing reduced precision neural network models.