Attention Mechanism Python Packages

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

vit-pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

linformer

Implementation of Linformer for Pytorch

flamingo-pytorch

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

reformer-pytorch

Reformer, the efficient Transformer, in Pytorch

tab-transformer-pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

zetascale

Build high-performance AI models with modular building blocks

keras-self-attention

Attention mechanism for processing sequential data that considers the context for each timestamp.

enformer-pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

en-transformer

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

alphafold2-pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

activations

AReLU: Attention-based-Rectified-Linear-Unit

x-transformers-rl

Implementation of a transformer for reinforcement learning using `x-transformers`

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

tf-bind-transformer

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

se3-transformer-pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

improving-transformers-world-model

Implementation of the new SOTA for model based RL, from the paper "Improving Transformer World Models for Data-Efficient RL", in Pytorch

nystrom-attention

Implementation of Nyström Self-attention, from the paper Nyströmformer

memorizing-transformers-pytorch

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

coca-pytorch

Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch