Attention Mechanisms Python Packages

local-attention

An implementation of local windowed attention for language modeling

colt5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

alphafold3-pytorch

Implementation of Alphafold 3 from Google Deepmind in Pytorch

audiolm-pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

meshgpt-pytorch

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

palm-rlhf-pytorch

PaLM + Reinforcement Learning with Human Feedback - Pytorch

bs-roformer

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

megabyte-pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

alphagenome-pytorch

Implementation of AlphaGenome, Deepmind's updated genomic attention model

magvit2-pytorch

Implementation of MagViT2 Tokenizer in Pytorch

calm-pytorch

CALM - Pytorch

tri-lbm

Implementation of the Large Behavioral Model architecture for dexterous manipulation from Toyota Research Institute

block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

coconut-pytorch

Coconut in Pytorch

itransformer

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

q-transformer

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

locoformer

LocoFormer - Generalist Locomotion via Long-Context Adaptation

recurrent-memory-transformer-pytorch

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

robotic-transformer-pytorch

Implementation of RT1 (Robotic Transformer) in Pytorch

simple-hierarchical-transformer

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

taylor-series-linear-attention

Explorations into the recently proposed Taylor Series Linear Attention

muse-maskgit-pytorch

MUSE - Text-to-Image Generation via Masked Generative Transformers, in Pytorch

phenaki-pytorch

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

mega-pytorch

Mega - Pytorch