Attention Mechanisms Python Packages

local-attention

An implementation of local windowed attention for language modeling

colt5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

megabyte-pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

alphafold3-pytorch

Implementation of Alphafold 3 from Google Deepmind in Pytorch

audiolm-pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

bs-roformer

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

palm-rlhf-pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

locoformer

LocoFormer - Generalist Locomotion via Long-Context Adaptation

meshgpt-pytorch

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

magvit2-pytorch

Implementation of MagViT2 Tokenizer in Pytorch

calm-pytorch

Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind

taylor-series-linear-attention

Explorations into the recently proposed Taylor Series Linear Attention

tri-lbm

Implementation of the Large Behavioral Model architecture for dexterous manipulation from Toyota Research Institute

q-transformer

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

phenaki-pytorch

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

simple-hierarchical-transformer

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

flash-cosine-sim-attention

Implementation of fused cosine similarity attention in the same style as Flash Attention

recurrent-memory-transformer-pytorch

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

itransformer

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

coconut-pytorch

Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch

dynastes

A collection of layers, ops, utilities and more for TensorFlow 2.0 high-level API Keras

muse-maskgit-pytorch

Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

musiclm-pytorch

Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch