Transformers Python Packages

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

peft

🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.

ultralytics-thop

Profile PyTorch models for FLOPs and parameters, helping to evaluate computational efficiency and memory usage.

presidio-analyzer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

presidio-anonymizer

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

mleap

MLeap: Deploy ML Pipelines to Production

optimum

🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

spark-nlp

State of the Art Natural Language Processing

haystack-ai

Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.

segmentation-models-pytorch

Semantic segmentation models with 500+ pretrained convolutional and transformer-based backbones.

ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

chronos-forecasting

Chronos: Pretrained Models for Time Series Forecasting

curated-transformers

🤖 A PyTorch library of curated Transformer models and their composable components

nncf

Neural Network Compression Framework for enhanced OpenVINO™ inference

bertopic

Leveraging BERT and c-TF-IDF to create easily interpretable topics.

llm-guard

The Security Toolkit for LLM Interactions

needle-python

Needle simplifies building RAG pipelines.

llamafactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

argostranslate

Open-source offline translation library written in Python

swanlab

⚡️SwanLab - an open-source, modern-design AI training tracking and visualization tool. Supports Cloud / Self-hosted use. Integrated with PyTorch / Transformers / verl / LLaMA Factory / ms-swift / Ultralytics / MMEngine / Keras etc.

mlx-audio

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple Silicon.

megatron-core

Ongoing research training transformer models at scale