Attention Mechanism Python Packages

x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers

vit-pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

linformer

Implementation of Linformer for Pytorch

enformer-pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

flamingo-pytorch

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

nystrom-attention

Implementation of Nyström Self-attention, from the paper Nyströmformer

reformer-pytorch

Reformer, the efficient Transformer, in Pytorch

performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

tab-transformer-pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

zetascale

Build high-performance AI models with modular building blocks

keras-self-attention

Attention mechanism for processing sequential data that considers the context for each timestamp.

en-transformer

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

flash-pytorch

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

alphafold2-pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

improving-transformers-world-model

Implementation of the new SOTA for model based RL, from the paper "Improving Transformer World Models for Data-Efficient RL", in Pytorch

se3-transformer-pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.

x-transformers-rl

Implementation of a transformer for reinforcement learning using `x-transformers`

tf-bind-transformer

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

activations

AReLU: Attention-based-Rectified-Linear-Unit

memorizing-transformers-pytorch

Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

pynabled

nabled aims to be Rust's goto library for Linear Algebra & ML: performant, comprehensive, and correct.