attention-mechanisms

Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM

An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

Implementation of Alphafold 3 from Google Deepmind in Pytorch

Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs

Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch

Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch

Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

PyTorch Dual-Attention LSTM-Autoencoder For Multivariate Time Series

Implementation of MagViT2 Tokenizer in Pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

Implementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorch

🦖Pytorch implementation of popular Attention Mechanisms, Vision Transformers, MLP-Like models and CNNs.🔥🔥🔥

Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group

An implementation of local windowed attention for language modeling