#Attention-mechanism
Showing 60 of 173 repositories tagged #attention-mechanism, ranked by stars
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch
邱锡鹏《神经网络与深度学习》（蒲公英书）理论书 v2 与通识版
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
all kinds of text classification models and more with deep learning
A concise but complete full-attention transformer with a set of promising experimental features from various papers
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites
RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.
Graph Attention Networks (https://arxiv.org/abs/1710.10903)
Automatic Speech Recognition (ASR), Speaker Verification, Speech Synthesis, Text-to-Speech (TTS), Language Modelling, Singing Voice Synthesis (SVS), Voice Conversion (VC)
Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)
Show, Attend, and Tell | a PyTorch Tutorial to Image Captioning
Multilingual Automatic Speech Recognition with word-level timestamps and confidence
Keras Attention Layer (Luong and Bahdanau scores).
My implementation of the original GAT paper (Veličković et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!
Build a modern LLM from scratch. Every line commented. Explained like we are five.
Reformer, the efficient Transformer, in Pytorch
To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released
Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch
Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute
Learn LLM internals step by step - from tokenization to attention to inference optimization.
Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch
Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch
Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.
Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch
An implementation of Performer, a linear attention-based transformer, in Pytorch
A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.
OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning
My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.
Implementation of TabTransformer, attention network for tabular data, in Pytorch
[CVPR2022] Geometric Transformer for Fast and Robust Point Cloud Registration
pytorch implementation of "Get To The Point: Summarization with Pointer-Generator Networks"
Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch
Transformer based on a variant of attention that is linear complexity in respect to sequence length
Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways
A PyTorch implementation of "SimGNN: A Neural Network Approach to Fast Graph Similarity Computation" (WSDM 2019).
This repository contains my full work and notes on Coursera's NLP Specialization (Natural Language Processing) taught by the instructor Younes Bensouda Mourri and Łukasz Kaiser offered by deeplearning.ai
Visualizing RNNs using the attention mechanism
Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification
[ECCV 2020] Learning Enriched Features for Real Image Restoration and Enhancement. SOTA results for image denoising, super-resolution, and image enhancement.
A Pytorch Implementation of "Neural Speech Synthesis with Transformer Network"
Implementation of Bottleneck Transformer in Pytorch
Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch
FlashAttention (Metal Port)
[IGARSS'22]: A Transformer-Based Siamese Network for Change Detection
Implementation of the Point Transformer layer, in Pytorch
Build high-performance AI models with modular building blocks
[ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
Democratization of RT-2 "RT-2: New model translates vision and language into action"
Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch
Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch
Implementation of Parti, Google's pure attention-based text-to-image neural network, in Pytorch
Neural Machine Translation with Keras
The official PyTorch implementation of the paper "SAITS: Self-Attention-based Imputation for Time Series". A fast and state-of-the-art (SOTA) deep-learning neural network model for efficient time-series imputation (impute multivariate incomplete time series containing NaN missing data/values with machine learning). https://arxiv.org/abs/2202.08516
A Structured Self-attentive Sentence Embedding
Implementation of Slot Attention from GoogleAI
A novel implementation of fusing ViT with Mamba into a fast, agile, and high performance Multi-Modal Model. Powered by Zeta, the simplest AI framework ever.
Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT.
a collection of visualization function