Flash Attention Python Packages

nvidia-cudnn-frontend

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

mlx-kquant

Native K-quant support for MLX, with a quantization and fine-tuning toolchain for Apple Silicon

ffpa-attn

Kernel Library for Large Headdim Attention (64~1024, BF16/FP8/FP4), 1.5x~15x↑ vs PyTorch SDPA.

mlx-train-perf

Fused, logit-free linear-cross-entropy loss, RAM-fit planner, and benchmark harness for MLX fine-tuning on Apple Silicon

flash-sinkhorn

The official repository of FlashSinkhorn [ICML 2026 Oral]

whisper-blaze

High-throughput batched Whisper large-v3 serving on NVIDIA Hopper GPUs — OpenAI-compatible API, dynamic batching, VRAM capping, fused Hopper LayerNorm kernel

inf-cl

[CVPR 2025 Highlight] The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.

flashslice

Fused Triton kernels for Transolver's slice/deslice. Official code for "Does Transolver really need a Transformer?" (arXiv:2609.32525)

axiomlm

You import the model's weights and start fine-tuning, unaware that you are missing out on more than 80% of gpu utilization. Axiom gets all the juice out of the box.

morphottention

Mathematical Morphology-based self-attention module for PyTorch (CUDA) using Flash-style kernel fusion.

flash-sparse-attn

Trainable fast and memory-efficient sparse attention

mlsys-lab

Auto-graded exercises in low-level ML systems: real C++, real CUDA-C on a software GPU (no NVIDIA hardware), deterministic measured gates.

darkformer-pytorch

Unofficial Pytorch-based implementation of Google DeepMind's "Data-Aware Random Feature Kernels for Transformers" architecture. Featuring encoder-decoder models, language models, deterministic projections, and optional FlashAttention 2/3 support.

flashmha

An simple pytorch implementation of Flash MultiHead Attention

rigsolve

Explainable compatibility resolution for torch, CUDA, and native GPU extensions.

gpkg

GPU package manager — find prebuilt CUDA wheels, build missing ones

fastflash-attention

Drop-in exact bf16 flash-attention for CUDA with a deterministic backward, tuned for Blackwell (sm_120 / RTX 5090).

flash-dmattn

Trainable fast and memory-efficient sparse attention

flash-attention-triton

Cross-platform FlashAttention-2 Triton implementation for Turing+ GPUs with custom configuration mode

easywheels

Smart GPU wheel installer. Auto-detects CUDA, GPU, torch, and Python.

jax-flash-attn2

A flexible and efficient implementation of Flash Attention 2.0 for JAX, supporting multiple backends (GPU/TPU/CPU) and platforms (Triton/Pallas/JAX).

scree

Variable-length tensors with Triton kernels at 1.6× of FlashAttention-2 on H100. NumPy / PyTorch / MLX / JAX backends.