#Optimizers

labmlai

annotated_deep_learning_paper_implementations

🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

lucidrains

lion-pytorch

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

elixir-nx

axon

Nx-powered Neural Networks

gugarosa

opytimizer

🐦 Opytimizer is a Python library consisting of meta-heuristic optimization algorithms.

Rishit-dagli

Gradient-Centralization-TensorFlow

Instantly improve your optimizer with just 2 lines of code!

LoganBooker

prodigy-plus-schedule-free

Prodigy and Schedule-Free, together at last.

MatthewK78

Rose

🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.

ShivamShrirao

dnn_from_scratch

A high level deep learning library for Convolutional Neural Networks,GANs and more, made from scratch(numpy/cupy implementation).