Transformers Python Packages

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

ultralytics-thop

Profile PyTorch models for FLOPs and parameters, helping to evaluate computational efficiency and memory usage.

presidio-analyzer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

presidio-anonymizer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

mleap

MLeap: Deploy ML Pipelines to Production

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

spark-nlp

State of the Art Natural Language Processing

haystack-ai

Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

segmentation-models-pytorch

Semantic segmentation models with 500+ pretrained convolutional and transformer-based backbones.

curated-transformers

🤖 A PyTorch library of curated Transformer models and their composable components

bertopic

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

chronos-forecasting

Chronos: Pretrained Models for Time Series Forecasting

ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

mlx-audio

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

transformer-smaller-training-vocab

Temporary remove unused tokens during training to save ram and speed.

swanlab

⚡️SwanLab - an open-source, modern-design AI training tracking and visualization tool. Supports Cloud / Self-hosted use. Integrated with PyTorch / Transformers / verl / LLaMA Factory / ms-swift / Ultralytics / MMEngine / Keras etc.

llm-guard

The Security Toolkit for LLM Interactions

argostranslate

Open-source offline translation library written in Python

needle-python

Needle simplifies building RAG pipelines.