Attention Mechanisms Python Packages

local-attention

An implementation of local windowed attention for language modeling

colt5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

alphafold3-pytorch

Implementation of Alphafold 3 from Google Deepmind in Pytorch

audiolm-pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

meshgpt-pytorch

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

bs-roformer

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

palm-rlhf-pytorch

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

magvit2-pytorch

Implementation of MagViT2 Tokenizer in Pytorch

alphagenome-pytorch

Implementation of AlphaGenome, Deepmind's updated genomic attention model

q-transformer

Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind

itransformer

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

block-recurrent-transformer-pytorch

Implementation of Block Recurrent Transformer - Pytorch

megabyte-pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

recurrent-memory-transformer-pytorch

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

locoformer

LocoFormer - Generalist Locomotion via Long-Context Adaptation

calm-pytorch

Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind

robotic-transformer-pytorch

Implementation of RT1 (Robotic Transformer) in Pytorch

simple-hierarchical-transformer

Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT

phenaki-pytorch

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

complex-valued-transformer

Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"

tri-lbm

Implementation of the Large Behavioral Model architecture for dexterous manipulation from Toyota Research Institute

coconut-pytorch

Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch

taylor-series-linear-attention

Explorations into the recently proposed Taylor Series Linear Attention

musiclm-pytorch

Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch