#Efficient-attention

thu-ml

SageAttention

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

lucidrains

CoLT5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

jlamprou

Infini-Attention

Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval

davidsvy

cosformer-pytorch

Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".