#Attention-mechanisms
Showing 50 of 50 repositories tagged #attention-mechanisms, ranked by stars
Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM
An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites
Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch
Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch
Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI
Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch
Implementation of Alphafold 3 from Google Deepmind in Pytorch
Awesome List of Attention Modules and Plug&Play Modules in Computer Vision
Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch
Implementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labs
Implementation of MeshGPT, SOTA Mesh generation using Attention, in Pytorch
Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch
Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
Implementation of MagViT2 Tokenizer in Pytorch
Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch
Implementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorch
Unofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant group
An implementation of local windowed attention for language modeling
Implementation of RT1 (Robotic Transformer) in Pytorch
Official PyTorch Implementation for "Rotate to Attend: Convolutional Triplet Attention Module." [WACV 2021]
Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch
Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind
Explorations into training LLMs to use clinical calculators from patient history, using open sourced models. Will start with Wells' Criteria
Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding
Neat (Neural Attention) Vision, is a visualization tool for the attention mechanisms of deep-learning models for Natural Language Processing (NLP) tasks. (framework-agnostic)
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
Implementation of Flash Attention in Jax
Implementation of Block Recurrent Transformer - Pytorch
Integrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modeling
Implementation of fused cosine similarity attention in the same style as Flash Attention
Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch
Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena
Implementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorch
Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
LocoFormer - Generalist Locomotion via Long-Context Adaptation
Explorations into the recently proposed Taylor Series Linear Attention
Implementation of Zorro, Masked Multimodal Transformer, in Pytorch
Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with Sparse Transformers"
Implementation of Agent Attention in Pytorch
Implementation of the transformer proposed in "Building Blocks for a Complex-Valued Transformer Architecture"
Multi heads attention for image classification
Implementation of Transframer, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch
Explorations into whether a transformer with RL can direct a genetic algorithm to converge faster
Implementation of Denoising Diffusion for protein design, but using the new Equiformer (successor to SE3 Transformers) with some additional improvements
My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)
Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token
Implementation of an Attention layer where each head can attend to more than just one token, using coordinate descent to pick topk
CUDA implementation of autoregressive linear attention, with all the latest research findings