Attention Mechanism Python Packages

whisper-timestamped

Multilingual Automatic Speech Recognition with word-level timestamps and confidence

linear-attention-transformer

Transformer based on a variant of attention that is linear complexity in respect to sequence length

linformer

Implementation of Linformer for Pytorch

enformer-pytorch

Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch

keras-self-attention

Attention mechanism for processing sequential data that considers the context for each timestamp.

reformer-pytorch

Reformer, the efficient Transformer, in Pytorch

zetascale

Build high-performance AI models with modular building blocks

recurrent-transformer-pytorch

Explorations into the Recurrent Transformer proposed by Costin-Andrei Oncescu et al. of Harvard University

alphafold2-pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

en-transformer

Implementation of E(n)-Transformer, which incorporates attention mechanisms into Welling's E(n)-Equivariant Graph Neural Network

nystrom-attention

Implementation of Nyström Self-attention, from the paper Nyströmformer

performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

mimic-video

Implementation of Mimic-Video, Video-Action Models for SOTA Generalizable Robot Control Beyond VLAs

tab-transformer-pytorch

Implementation of TabTransformer, attention network for tabular data, in Pytorch

dalle-pytorch

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

flamingo-pytorch

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

declarative-attention

Implementation of the procedure in Language Models Can Control Their Own Attention, from Namgyu Ho et al. of KAIST AI

sockeye

Sequence-to-sequence framework with a focus on Neural Machine Translation based on PyTorch

pynabled

nabled aims to be Rust's goto library for Linear Algebra & ML: performant, comprehensive, and correct.

tf-bind-transformer

A repository with exploration into using transformers to predict DNA ↔ transcription factor binding

ruvector

RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.

se3-transformer-pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication.