#Attention
Showing 60 of 96 repositories tagged #attention, ranked by stars
๐งโ๐ซ 60+ Implementations/tutorials of deep learning papers with side-by-side notes ๐; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), ๐ฎ reinforcement learning (ppo, dqn), capsnet, distillation, ... ๐ง
SGLang is a high-performance serving framework for large language models and multimodal models.
Machine learning, in numpy
A theoretical reconstruction of the Claude Mythos architecture, built from first principles using the available research literature.
๐ Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.โญโญโญ
A PyTorch implementation of the Transformer model in "Attention is All You Need".
FlashInfer: Kernel Library for LLM Serving
Scenic: A Jax Library for Computer Vision Research and Beyond
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
My implementation of the original GAT paper (Veliฤkoviฤ et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!
Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0
Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute
Improving Convolutional Networks via Attention Transfer (ICLR 2017)
Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI
Replication of simple CV Projects including attention, classification, detection, keypoint detection, etc.
Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.
An implementation of Performer, a linear attention-based transformer, in Pytorch
My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.
[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.
Simple implementations of NLP models. Tutorials are written in Chinese on my website https://mofanpy.com
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.
Official Pytorch Code for "Medical Transformer: Gated Axial-Attention for Medical Image Segmentation" - MICCAI 2021
Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper
Long Range Arena for Benchmarking Efficient Transformers
Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (NeurIPS 2019)
[CVPR 2023] Official code release of our paper "BiFormer: Vision Transformer with Bi-Level Routing Attention"
Julia Implementation of Transformer models
Implementation of MusicLM, a text to music model published by Google Research, with a few modifications.
The official PyTorch implementation of the paper "SAITS: Self-Attention-based Imputation for Time Series". A fast and state-of-the-art (SOTA) deep-learning neural network model for efficient time-series imputation (impute multivariate incomplete time series containing NaN missing data/values with machine learning). https://arxiv.org/abs/2202.08516
A Structured Self-attentive Sentence Embedding
a collection of visualization function
Keras, PyTorch, and NumPy Implementations of Deep Learning Architectures for NLP
BERT-NER (nert-bert) with google bert https://github.com/google-research.
Implementation of the ScreenAI model from the paper: "A Vision-Language Model for UI and Infographics Understanding"
ไธญๆ่ๅคฉๆบๅจไบบ๏ผๅบไบ10ไธ็ปๅฏน็ฝ่ฎญ็ป่ๆ๏ผ้็จๆณจๆๅๆบๅถ๏ผๅฏนไธ่ฌ้ฎ้ข้ฝไผ็ๆไธไธชๆๆไน็็ญๅคใๅทฒไธไผ ๆจกๅ๏ผๅฏ็ดๆฅ่ฟ่กใ
Traffic prediction is the task of predicting future traffic measurements (e.g. volume, speed, etc.) in a road network (graph), using historical data (timeseries).
Implementation of Spear-TTS - multi-speaker text-to-speech attention network, in Pytorch
PyTorch Implementation of Compact Generalized Non-local Network [NIPS'18]
Neat (Neural Attention) Vision, is a visualization tool for the attention mechanisms of deep-learning models for Natural Language Processing (NLP) tasks. (framework-agnostic)
Contains implementation of Guided Attention Inference Network (GAIN) presented in Tell Me Where to Look(CVPR 2018). This repository aims to apply GAIN on fcn8 architecture used for segmentation.
๐ฅ๐ฅ๐ฅA collection of some awesome public SNN(Spiking Neural Network) projects.
pretrained BERT model for cyber security text, learned CyberSecurity Knowledge
Implementation of Danijar's latest iteration for his Dreamer line of work
Deep-learning model presented in "DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis".
Implementation of the Llama architecture with RLHF + Q-learning
Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning
Implementation of a framework for Genie2 in Pytorch
Easy to use Python command line based tool to generate a gaze point heatmap from a csv file. ๐๏ธ
BertQA - Attention on Steroids
A hands-on course for building modern LLMs from scratch in PyTorch, with 26 runnable Jupyter Notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation, and distillation.
Single-layer transformer in HyperTalk for the classic Macintosh
Monitor Valorant gameplay through an external overlay to track player stats and match data in real time.
Compound Multi-branch Feature Fusion for Real Image Restoration
Code for AAAI 2022 Oral paper: 'Meta Faster R-CNN: Towards Accurate Few-Shot Object Detection with Attentive Feature Alignment'
Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval
My Attempt(s) In The World Of ML/DL....
[TIP'21] Learning Deep Global Multi-scale and Local Attention Features for Facial Expression Recognition in the Wild
Real-time JavaScript/WebGL library to detect whether the user is looking at the screen. Lightweight, robust to varied lighting, perfect for auto play-pause or presence-aware web experiences ๐