BossNAS
This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search. [pdf link]
Illustration of the Siamese supernets training with ensemble bootstrapping.
Illustration of the fabric-like Hybrid CNN-transformer Search Space with flexible down-sampling positions.
Our Results and Trained Models
-
Here is a summary of our searched models:
Model MAdds Steptime Top-1 (%) Top-5 (%) Url BossNet-T0 w/o SE 3.4B 101ms 80.5 95.0 checkpoint BossNet-T0 3.4B 115ms 80.8 95.2 checkpoint BossNet-T0^ 5.7B 147ms 81.6 95.6 same as above BossNet-T1 7.9B 156ms 81.9 95.6 checkpoint BossNet-T1^ 10.5B 165ms 82.2 95.7 same as above -
Here is a summary of architecture rating accuracy of our method:
Search space Dataset Kendall tau Spearman rho Pearson R MBConv ImageNet 0.65 0.78 0.85 NATS-Bench Ss Cifar10 0.53 0.73 0.72 NATS-Bench Ss Cifar100 0.59 0.76 0.79
Usage
1. Requirements
- Install PyTorch 1.7.0+ and torchvision 0.8.1+, for example:
conda install -c pytorch pytorch torchvision
- Install pytorch-image-models 0.3.2, for example:
pip install timm==0.3.2
-
Download the ImageNet dataset from http://image-net.org/, and move validation images to labeled subfolders
- To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh
2. Retrain or Evaluate our BossNet-T models
-
First, move to retraining code directory to perform Retraining or Evaluation.
cd HyTra_retrainingOur retraining code of BossNet-T is based on DeiT repository.
-
You can evaluate our BossNet-T models with the following command:
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T0-80_8.pth
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T1-81_9.pth
Please download our checkpoint files from the result table. Please change the
--nproc_per_nodeoption to suit your GPU numbers, and change the--data-path,--resumeand--input-sizeaccordingly. -
You can retrain our BossNet-T models with the following command:
Please change the
--nproc_per_nodeand--data-pathaccordingly. Note that the learning rate will be automatically scaled according to the GPU numbers and batchsize. We recommend training with 128 batchsize and 8 GPUs.python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
Architecture of our BossNet-T0
3. Evaluate architecture rating accuracy of BossNAS
-
You can get the ranking correlations of BossNAS on MBConv search space with the following commands:
cd MBConv_ranking python get_model_score_mbconv.py
- You can get the ranking correlations of BossNAS on NATS-Bench Ss with the following commands:
cd NATS_SS_ranking python get_model_score_nats.py
Citation
@article{li2021bossnas,
author = {Li, Changlin and
Tang, Tao and
Wang, Guangrun and
Peng, Jiefeng and
Wang, Bing and
Liang, Xiaodan and
Chang, Xiaojun},
title = {BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search},
journal = {arXiv:2103.12424},
year = 2021,
}
TODO
Searching code will be released later.
2 Nov 28, 2021
5 Nov 08, 2022
60 Jan 04, 2023
32 Dec 13, 2022
0 Oct 31, 2021
1.3k Dec 28, 2022
27 Dec 06, 2022
4 Dec 14, 2022
125 Dec 06, 2022
0 Jul 14, 2021
221 Jan 07, 2023
135 Dec 27, 2022
22 Jul 20, 2022
17 Jul 04, 2022
40 Dec 12, 2022
62 Dec 03, 2022
4.6k Dec 30, 2022
2 Dec 27, 2022
30 Jul 14, 2022
22 Dec 31, 2022