PyTorch Image Models

• What's New
• Introduction
• Models
• Features
• Results
• Getting Started (Documentation)
• Train, Validation, Inference Scripts
• Awesome PyTorch Resources
• Licenses
• Citing

What's New

May 27, 2026

• Add model defs and pretrained weights for EUPE ViT (DINOv3-style) and ConvNeXt models. See the Efficient Universal Perception Encoder paper.
• Add TIPSv2 model defs and pretrained weights for (DINOv2-style) ViTs. See the TIPSv2 paper.

May 8, 2026

• Release 1.0.27

April 23, 2026

• Add Gemma4 ViT encoders w/ NaFlex pipeline support (variable aspect/size per image). Thanks Yonghye Kwon
• Support DINOv3 weights in NaFlexVit. Thanks Yonghye Kwon
• Some improvements to Muon fallback (AdamW/NadamW) lr behavior

March 23, 2026

• Improve pickle checkpoint handling security. Default all loading to weightsonly=True, add safeglobal for ArgParse.
• Improve attention mask handling for core ViT/EVA models & layers. Resolve bool masks, pass is_causal through for SSL tasks.
• Fix class & register token uses with ViT and no pos embed enabled.
• Add Patch Representation Refinement (PRR) as a pooling option in ViT. Thanks Sina (https://github.com/sinahmr).
• Improve consistency of output projection / MLP dimensions for attention pooling layers.
• Hiera model F.SDPA optimization to allow Flash Attention kernel use.
• Caution added to SGDP optimizer.
• Release 1.0.26. First maintenance release since my departure from Hugging Face.

Feb 23, 2026

• Add token distillation training support to distillation task wrappers
• Remove some torch.jit usage in prep for official deprecation
• Caution added to AdamP optimizer
• Call resetparameters() even if meta-device init so that buffers get init w/ hacks like initempty_weights
• Tweak Muon optimizer to work with DTensor/FSDP2 (clamp instead of clampmin_, alternate NS branch for DTensor)
• Release 1.0.25

Jan 21, 2026

• Compat Break: Fix oversight w/ QKV vs MLP bias in ParallelScalingBlock (& DiffParallelScalingBlock)

Jan 5 & 6, 2026

• Release 1.0.24
• Add new benchmark result csv files for inference timing on all models w/ RTX Pro 6000, 5090, and 4090 cards w/ PyTorch 2.9.1
• Fix moved module error in deprecated timm.models.layers import path that impacts legacy imports
• Release 1.0.23

Dec 30, 2025

• Add better NAdaMuon trained dpwee, dwee, dlittle (differential) ViTs with a small boost over previous runs

• Add a ~21M param timm variant of the CSATv2 model at 512x512 & 640x640

• Factor non-persistent param init out of init into a common method that can be externally called via initnonpersistent_buffers() after meta-device init.

Dec 12, 2025

• Add CSATV2 model (thanks https://github.com/gusdlf93) -- a lightweight but high res model with DCT stem & spatial attention. https://huggingface.co/Hyunil/CSATv2
• Add AdaMuon and NAdaMuon optimizer support to existing timm Muon impl. Appears more competitive vs AdamW with familiar hparams for image tasks.
• End of year PR cleanup, merge aspects of several long open PR

• Bump unit tests to PyTorch 2.9.1 + Python 3.13 on upper end, lower still PyTorch 1.13 + Python 3.10

Dec 1, 2025

• Add lightweight task abstraction, add logits and feature distillation support to train script via new tasks.
• Remove old APEX AMP support

Nov 4, 2025

• Fix LayerScale / LayerScale2d init bug (init values ignored), introduced in 1.0.21. Thanks https://github.com/Ilya-Fradlin
• Release 1.0.22

Oct 31, 2025 🎃

• Update imagenet & OOD variant result csv files to include a few new models and verify correctness over several torch & timm versions
• EfficientNet-X and EfficientNet-H B5 model weights added as part of a hparam search for AdamW vs Muon (still iterating on Muon runs)

Oct 16-20, 2025

• Add an impl of the Muon optimizer (based on https://github.com/KellerJordan/Muon) with customizations

• First 2 steps of 'meta' device model initialization supported

• License fields added to pretrained cfgs in code
• Release 1.0.21

Sept 21, 2025

• Remap DINOv3 ViT weight tags from lvd1689m -> lvd1689m to match (same for sat493m -> sat493m)
• Release 1.0.20

Sept 17, 2025

• DINOv3 (https://arxiv.org/abs/2508.10104) ConvNeXt and ViT models added. ConvNeXt models were mapped to existing timm model. ViT support done via the EVA base model w/ a new RotaryEmbeddingDinoV3 to match the DINOv3 specific RoPE impl

• MobileCLIP-2 (https://arxiv.org/abs/2508.20691) vision encoders. New MCI3/MCI4 FastViT variants added and weights mapped to existing FastViT and B, L/14 ViTs.
• MetaCLIP-2 Worldwide (https://arxiv.org/abs/2507.22062) ViT encoder weights added.
• SigLIP-2 (https://arxiv.org/abs/2502.14786) NaFlex ViT encoder weights added via timm NaFlexViT model.
• Misc fixes and contributions

July 23, 2025

• Add setinputsize() method to EVA models, used by OpenCLIP 3.0.0 to allow resizing for timm based encoder models.
• Release 1.0.18, needed for PE-Core S & T models in OpenCLIP 3.0.0
• Fix small typing issue that broke Python 3.9 compat. 1.0.19 patch release.

July 21, 2025

• ROPE support added to NaFlexViT. All models covered by the EVA base (eva.py) including EVA, EVA02, Meta PE ViT, timm SBB ViT w/ ROPE, and Naver ROPE-ViT can be now loaded in NaFlexViT when use_naflex=True passed at model creation time
• More Meta PE ViT encoders added, including small/tiny variants, lang variants w/ tiling, and more spatial variants.
• PatchDropout fixed with NaFlexViT and also w/ EVA models (regression after adding Naver ROPE-ViT)
• Fix XY order with grid_indexing='xy', impacted non-square image use in 'xy' mode (only ROPE-ViT and PE impacted).

July 7, 2025

• MobileNet-v5 backbone tweaks for improved Google Gemma 3n behaviour (to pair with updated official weights)

• Add two arguments to layer-decay support, a min scale clamp and 'no optimization' scale threshold
• Add 'Fp32' LayerNorm, RMSNorm, SimpleNorm variants that can be enabled to force computation of norm in float32
• Some typing, argument cleanup for norm, norm+act layers done with above
• Support Naver ROPE-ViT (https://github.com/naver-ai/rope-vit) in eva.py, add RotaryEmbeddingMixed module for mixed mode, weights on HuggingFace Hub

• Some cleanup of ROPE modules, helpers, and FX tracing leaf registration
• Preparing version 1.0.17 release

June 26, 2025

• MobileNetV5 backbone (w/ encoder only variant) for Gemma 3n image encoder
• Version 1.0.16 released

June 23, 2025

• Add F.grid_sample based 2D and factorized pos embed resize to NaFlexViT. Faster when lots of different sizes (based on example by https://github.com/stas-sl).
• Further speed up patch embed resample by replacing vmap with matmul (based on snippet by https://github.com/stas-sl).
• Add 3 initial native aspect NaFlexViT checkpoints created while testing, ImageNet-1k and 3 different pos embed configs w/ same hparams.

• Support gradient checkpointing for forward_intermediates and fix some checkpointing bugs. Thanks https://github.com/brianhou0208
• Add 'corrected weight decay' (https://arxiv.org/abs/2506.02285) as option to AdamW (legacy), Adopt, Kron, Adafactor (BV), Lamb, LaProp, Lion, NadamW, RmsPropTF, SGDW optimizers
• Switch PE (perception encoder) ViT models to use native timm weights instead of remapping on the fly
• Fix cuda stream bug in prefetch loader

June 5, 2025

• Initial NaFlexVit model code. NaFlexVit is a Vision Transformer with:

• Existing vit models in visiontransformer.py can be loaded into the NaFlexVit model by adding the usenaflex=True flag to create_model

• A full NaFlex data pipeline is available that allows training / fine-tuning / evaluating with variable aspect / size images

• To evaluate an existing (classic) ViT loaded in NaFlexVit model w/ NaFlex data pipe:

• The training has some extra args features worth noting

May 28, 2025

• Add a number of small/fast models thanks to https://github.com/brianhou0208

• Update EVA ViT (closest match) to support Perception Encoder models (https://arxiv.org/abs/2504.13181) from Meta, loading Hub weights but I still need to push dedicated timm weights

• Big increase in number of models supporting forward_intermediates() and some additional fixes thanks to https://github.com/brianhou0208

• TNT model updated w/ new weights forward_intermediates() thanks to https://github.com/brianhou0208
• Add local-dir: pretrained schema, can use local-dir:/path/to/model/folder for model name to source model / pretrained cfg & weights Hugging Face Hub models (config.json + weights file) from a local folder.
• Fixes, improvements for onnx export

Feb 21, 2025

• SigLIP 2 ViT image encoders added (https://huggingface.co/collections/timm/siglip-2-67b8e72ba08b09dd97aecaf9)

• Add 'SO150M2' ViT weights trained with SBB recipes, great results, better for ImageNet than previous attempt w/ less training.

• Updated InternViT-300M '2.5' weights
• Release 1.0.15

Feb 1, 2025

• FYI PyTorch 2.6 & Python 3.13 are tested and working w/ current main and released version of timm

Jan 27, 2025

• Add Kron Optimizer (PSGD w/ Kronecker-factored preconditioner)

Jan 19, 2025

• Fix loading of LeViT safetensor weights, remove conversion code which should have been deactivated
• Add 'SO150M' ViT weights trained with SBB recipes, decent results, but not optimal shape for ImageNet-12k/1k pretrain/ft

• Misc typing, typo, etc. cleanup
• 1.0.14 release to get above LeViT fix out

Jan 9, 2025

• Add support to train and validate in pure bfloat16 or float16
• wandb project name arg added by https://github.com/caojiaolong, use arg.experiment for name
• Fix old issue w/ checkpoint saving not working on filesystem w/o hard-link support (e.g. FUSE fs mounts)
• 1.0.13 release

Jan 6, 2025

• Add torch.utils.checkpoint.checkpoint() wrapper in timm.models that defaults usereentrant=False, unless TIMMREENTRANT_CKPT=1 is set in env.

Dec 31, 2024

• convnextnano 384x384 ImageNet-12k pretrain & fine-tune. https://huggingface.co/models?search=convnextnano%20r384
• Add AIM-v2 encoders from https://github.com/apple/ml-aim, see on Hub: https://huggingface.co/models?search=timm%20aimv2
• Add PaliGemma2 encoders from https://github.com/google-research/big_vision to existing PaliGemma, see on Hub: https://huggingface.co/models?search=timm%20pali2
• Add missing L/14 DFN2B 39B CLIP ViT, vitlargepatch14clip224.dfn2b_s39b
• Fix existing RmsNorm layer & fn to match standard formulation, use PT 2.5 impl when possible. Move old impl to SimpleNorm layer, it's LN w/o centering or bias. There were only two timm models using it, and they have been updated.
• Allow override of cache_dir arg for model creation
• Pass through trustremotecode for HF datasets wrapper
• inceptionnextatto model added by creator
• Adan optimizer caution, and Lamb decoupled weight decay options
• Some feature_info metadata fixed by https://github.com/brianhou0208
• All OpenCLIP and JAX (CLIP, SigLIP, Pali, etc) model weights that used load time remapping were given their own HF Hub instances so that they work with hf-hub: based loading, and thus will work with new Transformers TimmWrapperModel

Introduction

PyTorch Image Models (timm) is a collection of image models, layers, utilities, optimizers, schedulers, data-loaders / augmentations, and reference training / validation scripts that aim to pull together a wide variety of SOTA models with ability to reproduce ImageNet training results.

The work of many others is present here. I've tried to make sure all source material is acknowledged via links to github, arxiv papers, etc in the README, documentation, and code docstrings. Please let me know if I missed anything.

Features

Models

All model architecture families include variants with pretrained weights. There are specific model variants without any weights, it is NOT a bug. Help training new or better weights is always appreciated.

• Aggregating Nested Transformers - https://arxiv.org/abs/2105.12723
• BEiT - https://arxiv.org/abs/2106.08254
• BEiT-V2 - https://arxiv.org/abs/2208.06366
• BEiT3 - https://arxiv.org/abs/2208.10442
• Big Transfer ResNetV2 (BiT) - https://arxiv.org/abs/1912.11370
• Bottleneck Transformers - https://arxiv.org/abs/2101.11605
• CaiT (Class-Attention in Image Transformers) - https://arxiv.org/abs/2103.17239
• CoaT (Co-Scale Conv-Attentional Image Transformers) - https://arxiv.org/abs/2104.06399
• CoAtNet (Convolution and Attention) - https://arxiv.org/abs/2106.04803
• ConvNeXt - https://arxiv.org/abs/2201.03545
• ConvNeXt-V2 - http://arxiv.org/abs/2301.00808
• ConViT (Soft Convolutional Inductive Biases Vision Transformers)- https://arxiv.org/abs/2103.10697
• CspNet (Cross-Stage Partial Networks) - https://arxiv.org/abs/1911.11929
• DeiT - https://arxiv.org/abs/2012.12877
• DeiT-III - https://arxiv.org/pdf/2204.07118.pdf
• DenseNet - https://arxiv.org/abs/1608.06993
• DLA - https://arxiv.org/abs/1707.06484
• DPN (Dual-Path Network) - https://arxiv.org/abs/1707.01629
• EdgeNeXt - https://arxiv.org/abs/2206.10589
• EfficientFormer - https://arxiv.org/abs/2206.01191
• EfficientFormer-V2 - https://arxiv.org/abs/2212.08059
• EfficientNet (MBConvNet Family)

• EfficientViT (MIT) - https://arxiv.org/abs/2205.14756
• EfficientViT (MSRA) - https://arxiv.org/abs/2305.07027
• EVA - https://arxiv.org/abs/2211.07636
• EVA-02 - https://arxiv.org/abs/2303.11331
• FasterNet - https://arxiv.org/abs/2303.03667
• FastViT - https://arxiv.org/abs/2303.14189
• FlexiViT - https://arxiv.org/abs/2212.08013
• FocalNet (Focal Modulation Networks) - https://arxiv.org/abs/2203.11926
• GCViT (Global Context Vision Transformer) - https://arxiv.org/abs/2206.09959
• GhostNet - https://arxiv.org/abs/1911.11907
• GhostNet-V2 - https://arxiv.org/abs/2211.12905
• GhostNet-V3 - https://arxiv.org/abs/2404.11202
• gMLP - https://arxiv.org/abs/2105.08050
• GPU-Efficient Networks - https://arxiv.org/abs/2006.14090
• Halo Nets - https://arxiv.org/abs/2103.12731
• HGNet / HGNet-V2 - TBD
• HRNet - https://arxiv.org/abs/1908.07919
• InceptionNeXt - https://arxiv.org/abs/2303.16900
• Inception-V3 - https://arxiv.org/abs/1512.00567
• Inception-ResNet-V2 and Inception-V4 - https://arxiv.org/abs/1602.07261
• Lambda Networks - https://arxiv.org/abs/2102.08602
• LeViT (Vision Transformer in ConvNet's Clothing) - https://arxiv.org/abs/2104.01136
• MambaOut - https://arxiv.org/abs/2405.07992
• MaxViT (Multi-Axis Vision Transformer) - https://arxiv.org/abs/2204.01697
• MetaFormer (PoolFormer-v2, ConvFormer, CAFormer) - https://arxiv.org/abs/2210.13452
• MLP-Mixer - https://arxiv.org/abs/2105.01601
• MobileCLIP - https://arxiv.org/abs/2311.17049
• MobileNet-V3 (MBConvNet w/ Efficient Head) - https://arxiv.org/abs/1905.02244

• MobileNetV4 - https://arxiv.org/abs/2404.10518
• MobileOne - https://arxiv.org/abs/2206.04040
• MobileViT - https://arxiv.org/abs/2110.02178
• MobileViT-V2 - https://arxiv.org/abs/2206.02680
• MViT-V2 (Improved Multiscale Vision Transformer) - https://arxiv.org/abs/2112.01526
• NASNet-A - https://arxiv.org/abs/1707.07012
• NesT - https://arxiv.org/abs/2105.12723
• Next-ViT - https://arxiv.org/abs/2207.05501
• NFNet-F - https://arxiv.org/abs/2102.06171
• NF-RegNet / NF-ResNet - https://arxiv.org/abs/2101.08692
• PE (Perception Encoder) - https://arxiv.org/abs/2504.13181
• PNasNet - https://arxiv.org/abs/1712.00559
• PoolFormer (MetaFormer) - https://arxiv.org/abs/2111.11418
• Pooling-based Vision Transformer (PiT) - https://arxiv.org/abs/2103.16302
• PVT-V2 (Improved Pyramid Vision Transformer) - https://arxiv.org/abs/2106.13797
• RDNet (DenseNets Reloaded) - https://arxiv.org/abs/2403.19588
• RegNet - https://arxiv.org/abs/2003.13678
• RegNetZ - https://arxiv.org/abs/2103.06877
• RepVGG - https://arxiv.org/abs/2101.03697
• RepGhostNet - https://arxiv.org/abs/2211.06088
• RepViT - https://arxiv.org/abs/2307.09283
• ResMLP - https://arxiv.org/abs/2105.03404
• ResNet/ResNeXt

• Res2Net - https://arxiv.org/abs/1904.01169
• ResNeSt - https://arxiv.org/abs/2004.08955
• ReXNet - https://arxiv.org/abs/2007.00992
• ROPE-ViT - https://arxiv.org/abs/2403.13298
• SelecSLS - https://arxiv.org/abs/1907.00837
• Selective Kernel Networks - https://arxiv.org/abs/1903.06586
• Sequencer2D - https://arxiv.org/abs/2205.01972
• SHViT - https://arxiv.org/abs/2401.16456
• SigLIP (image encoder) - https://arxiv.org/abs/2303.15343
• SigLIP 2 (image encoder) - https://arxiv.org/abs/2502.14786
• StarNet - https://arxiv.org/abs/2403.19967
• SwiftFormer - https://arxiv.org/pdf/2303.15446
• Swin S3 (AutoFormerV2) - https://arxiv.org/abs/2111.14725
• Swin Transformer - https://arxiv.org/abs/2103.14030
• Swin Transformer V2 - https://arxiv.org/abs/2111.09883
• TinyViT - https://arxiv.org/abs/2207.10666
• Transformer-iN-Transformer (TNT) - https://arxiv.org/abs/2103.00112
• TResNet - https://arxiv.org/abs/2003.13630
• Twins (Spatial Attention in Vision Transformers) - https://arxiv.org/pdf/2104.13840.pdf
• VGG - https://arxiv.org/abs/1409.1556
• Visformer - https://arxiv.org/abs/2104.12533
• Vision Transformer - https://arxiv.org/abs/2010.11929
• ViTamin - https://arxiv.org/abs/2404.02132
• VOLO (Vision Outlooker) - https://arxiv.org/abs/2106.13112
• VovNet V2 and V1 - https://arxiv.org/abs/1911.06667
• Xception - https://arxiv.org/abs/1610.02357
• Xception (Modified Aligned, Gluon) - https://arxiv.org/abs/1802.02611
• Xception (Modified Aligned, TF) - https://arxiv.org/abs/1802.02611
• XCiT (Cross-Covariance Image Transformers) - https://arxiv.org/abs/2106.09681

Optimizers

Included optimizers available via timm.optim.createoptimizerv2 factory method:

• adabelief an implementation of AdaBelief adapted from https://github.com/juntang-zhuang/Adabelief-Optimizer - https://arxiv.org/abs/2010.07468
• adafactor adapted from FAIRSeq impl - https://arxiv.org/abs/1804.04235
• adafactorbv adapted from Big Vision - https://arxiv.org/abs/2106.04560
• adahessian by David Samuel - https://arxiv.org/abs/2006.00719
• adamp and sgdp by Naver ClovAI - https://arxiv.org/abs/2006.08217
• adamuon and nadamuon as per https://github.com/Chongjie-Si/AdaMuon - https://arxiv.org/abs/2507.11005
• adan an implementation of Adan adapted from https://github.com/sail-sg/Adan - https://arxiv.org/abs/2208.06677
• adopt ADOPT adapted from https://github.com/iShohei220/adopt - https://arxiv.org/abs/2411.02853
• kron PSGD w/ Kronecker-factored preconditioner from https://github.com/evanatyourservice/kron_torch - https://sites.google.com/site/lixilinx/home/psgd
• lamb an implementation of Lamb and LambC (w/ trust-clipping) cleaned up and modified to support use with XLA - https://arxiv.org/abs/1904.00962
• laprop optimizer from https://github.com/Z-T-WANG/LaProp-Optimizer - https://arxiv.org/abs/2002.04839
• lars an implementation of LARS and LARC (w/ trust-clipping) - https://arxiv.org/abs/1708.03888
• lion and implementation of Lion adapted from https://github.com/google/automl/tree/master/lion - https://arxiv.org/abs/2302.06675
• lookahead adapted from impl by Liam - https://arxiv.org/abs/1907.08610
• madgrad an implementation of MADGRAD adapted from https://github.com/facebookresearch/madgrad - https://arxiv.org/abs/2101.11075
• mars MARS optimizer from https://github.com/AGI-Arena/MARS - https://arxiv.org/abs/2411.10438
• muon MUON optimizer from https://github.com/KellerJordan/Muon with numerous additions and improved non-transformer behaviour
• nadam an implementation of Adam w/ Nesterov momentum
• nadamw an implementation of AdamW (Adam w/ decoupled weight-decay) w/ Nesterov momentum. A simplified impl based on https://github.com/mlcommons/algorithmic-efficiency
• novograd by Masashi Kimura - https://arxiv.org/abs/1905.11286
• radam by Liyuan Liu - https://arxiv.org/abs/1908.03265
• rmsprop_tf adapted from PyTorch RMSProp by myself. Reproduces much improved Tensorflow RMSProp behaviour
• sgdw and implementation of SGD w/ decoupled weight-decay
• fused<name> optimizers by name with NVIDIA Apex installed
• bnb<name> optimizers by name with BitsAndBytes installed
• cadamw, clion, and more 'Cautious' optimizers from https://github.com/kyleliang919/C-Optim - https://arxiv.org/abs/2411.16085
• adam, adamw, rmsprop, adadelta, adagrad, and sgd pass through to torch.optim implementations
• c suffix (eg adamc, nadamc to implement 'corrected weight decay' in https://arxiv.org/abs/2506.02285)

Augmentations

• Random Erasing from Zhun Zhong - https://arxiv.org/abs/1708.04896)
• Mixup - https://arxiv.org/abs/1710.09412
• CutMix - https://arxiv.org/abs/1905.04899
• AutoAugment (https://arxiv.org/abs/1805.09501) and RandAugment (https://arxiv.org/abs/1909.13719) ImageNet configurations modeled after impl for EfficientNet training (https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py)
• AugMix w/ JSD loss, JSD w/ clean + augmented mixing support works with AutoAugment and RandAugment as well - https://arxiv.org/abs/1912.02781
• SplitBachNorm - allows splitting batch norm layers between clean and augmented (auxiliary batch norm) data

Regularization

• DropPath aka "Stochastic Depth" - https://arxiv.org/abs/1603.09382
• DropBlock - https://arxiv.org/abs/1810.12890
• Blur Pooling - https://arxiv.org/abs/1904.11486

Other

Several (less common) features that I often utilize in my projects are included. Many of their additions are the reason why I maintain my own set of models, instead of using others' via PIP:

• All models have a common default configuration interface and API for

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}