SparK: the first successful BERT/MAE-style pretraining on any convolutional networks  

This is the official implementation of ICLR paper Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling, which can pretrain any CNN (e.g., ResNet) in a BERT-style self-supervised manner. We've tried our best to make the codebase clean, short, easy to read, state-of-the-art, and only rely on minimal dependencies.

https://user-images.githubusercontent.com/39692511/226858919-dd4ccf7e-a5ba-4a33-ab21-4785b8a7833c.mp4

🔥 News

• A brief introduction (in English) is available on our ICLR poster page! [📹Recorded Video, Poster, and Slides].
• On May. 11th another livestream on OpenMMLab & ReadPaper (bilibili)! [📹Recorded Video]
• On Apr. 27th (UTC+8 8pm) another livestream would be held at OpenMMLab (bilibili)!
• On Mar. 22nd (UTC+8 8pm) another livestream would be held at 极市平台 (bilibili)! [📹Recorded Video]
• The share on TechBeat (将门创投) is scheduled on Mar. 16th (UTC+8 8pm) too! [📹Recorded Video]
• We are honored to be invited by Synced ("机器之心机动组 视频号" on WeChat) to give a talk about SparK on Feb. 27th (UTC+0 11am, UTC+8 7pm), welcome! [📹Recorded Video]
• This work got accepted to ICLR 2023 as a Spotlight (notable-top-25%).
• Other articles: [Synced]

[DeepAI] [TheGradient] [Bytedance] [CVers [QbitAI(量子位)] [BAAI(智源)] [机器之心机动组] [极市平台] [ReadPaper笔记]

🕹️ Colab Visualization Demo

Check pretrain/viz_reconstruction.ipynb for visualizing the reconstruction of SparK pretrained models, like:

We also provide pretrain/viz_spconv.ipynb that shows the "mask pattern vanishing" issue of dense conv layers.

What's new here?

🔥 Pretrained CNN beats pretrained Swin-Transformer:

🔥 After SparK pretraining, smaller models can beat un-pretrained larger models:

🔥 All models can benefit, showing a scaling behavior:

🔥 Generative self-supervised pretraining surpasses contrastive learning:

See our paper for more analysis, discussions, and evaluations.

Todo list

catalog

• [x] Pretraining code
• [x] Pretraining toturial for customized CNN model (Tutorial for pretraining your own CNN model)
• [x] Pretraining toturial for customized dataset (Tutorial for pretraining your own dataset)
• [x] Pretraining Colab visualization playground (reconstruction, sparse conv)
• [x] Finetuning code
• [ ] Weights & visualization playground in huggingface
• [ ] Weights in timm

Pretrained weights (self-supervised; w/o decoder; can be directly finetuned)

Note: for network definitions, we directly use timm.models.ResNet and official ConvNeXt.

reso.: the image resolution; acc@1: ImageNet-1K finetuned acc (top-1)

| arch. | reso. | acc@1 | #params | flops | weights (self-supervised, without SparK's decoder) | |:--------------:|:-----:|:-----:|:-------:|:------:|:---------------------------------------------------------------------------------------------------------------------------------------| | ResNet50 | 224 | 80.6 | 26M | 4.1G | resnet501kpretrainedtimm_style.pth | | ResNet101 | 224 | 82.2 | 45M | 7.9G | resnet1011kpretrainedtimmstyle.pth | | ResNet152 | 224 | 82.7 | 60M | 11.6G | resnet1521kpretrainedtimm_style.pth | | ResNet200 | 224 | 83.1 | 65M | 15.1G | resnet2001kpretrainedtimmstyle.pth | | ConvNeXt-S | 224 | 84.1 | 50M | 8.7G | convnextS1kpretrainedofficial_style.pth | | ConvNeXt-B | 224 | 84.8 | 89M | 15.4G | convnextB1kpretrainedofficial_style.pth | | ConvNeXt-L | 224 | 85.4 | 198M | 34.4G | convnextL1kpretrainedofficialstyle.pth | | ConvNeXt-L | 384 | 86.0 | 198M | 101.0G | convnextL3841kpretrainedofficialstyle.pth |

Pretrained weights (with SparK's UNet-style decoder; can be used to reconstruct images)

| arch. | reso. | acc@1 | #params | flops | weights (self-supervised, with SparK's decoder) | |:----------:|:-----:|:-----:|:-------:|:------:|:------------------------------------------------------------------------------------------------------------------------------------------| | ResNet50 | 224 | 80.6 | 26M | 4.1G | res50withdecoder1kpretrainedsparkstyle.pth | | ResNet101 | 224 | 82.2 | 45M | 7.9G | res101withdecoder1kpretrainedsparkstyle.pth | | ResNet152 | 224 | 82.7 | 60M | 11.6G | res152withdecoder1kpretrainedsparkstyle.pth | | ResNet200 | 224 | 83.1 | 65M | 15.1G | res200withdecoder1kpretrainedspark_style.pth | | ConvNeXt-S | 224 | 84.1 | 50M | 8.7G | cnxS224withdecoder1kpretrainedsparkstyle.pth | | ConvNeXt-L | 384 | 86.0 | 198M | 101.0G | cnxL384withdecoder1kpretrainedspark_style.pth |

Installation & Running

We highly recommended you to use torch==1.10.0, torchvision==0.11.1, and timm==0.5.4 for reproduction. Check INSTALL.md to install all pip dependencies.

• Loading pretrained model weights in 3 lines

# download our weights resnet501kpretrainedtimm_style.pth first
import torch, timm
res50, state = timm.createmodel('resnet50'), torch.load('resnet501kpretrainedtimmstyle.pth', 'cpu')
res50.loadstatedict(state.get('module', state), strict=False)     # just in case the model weights are actually saved in state['module']

• Pretraining

- any ResNet or ConvNeXt on ImageNet-1k:  see pretrain/ - your own CNN model:  see pretrain/, especially pretrain/models/custom.py

• Finetuning

- any ResNet or ConvNeXt on ImageNet-1k:  check downstream_imagenet/ for subsequent instructions. - ResNets on COCO:  see downstream_d2/ - ConvNeXts on COCO:  see downstream_mmdet/

Acknowledgement

We referred to these useful codebases:

• BEiT, MAE, ConvNeXt
• timm, MoCoV2, Detectron2, MMDetection

License

This project is under the MIT license. See LICENSE for more details.

Citation

If you found this project useful, you can kindly give us a star ⭐, or cite us in your work 📖:

@Article{tian2023designing,   author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},   title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},   journal = {arXiv:2301.03580},   year    = {2023}, }