#Mixture-of-experts
Showing 23 of 23 repositories tagged #mixture-of-experts, ranked by stars
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Optimizing inference proxy for LLMs
Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
Run Mixtral-8x7B models in Colab or consumer desktops
⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)
Tutel MoE: Optimized Mixture-of-Experts Library, Support GptOss/DeepSeek/Kimi-K2/Qwen3 using FP8/NVFP4/MXFP4
SMT: The Surrogate Modeling Toolbox
cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.
A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)
A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)
中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
⚡ Zero-Stall MoE Inference via Lookahead Prediction & Async DMA Prefetching. Optimized for SSD I/O with Hybrid MLA+Sliding Window Attention.
The official implementation of the paper "Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques (TMLR)".
PyTorch implementation of Soft MoE by Google Brain in "From Sparse to Soft Mixtures of Experts" (https://arxiv.org/pdf/2308.00951.pdf)
A Step-by-Step Implementation of Qwen 3 MoE Architecture from Scratch
PyTorch implementation of "From Sparse to Soft Mixtures of Experts"
Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)
Nodes to run Hunyuan Image 3 locally with BF16 and NF4 quantized options in Comfyui
Toy-scale unified multimodal model experiments — encoder-free understanding & generation with Mixture-of-Transformers on MLX/Apple Silicon