code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Overview

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

Code for paper:

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, Mi Zhang.
NeurIPS 2020.

arch2vec
Top: The supervision signal for representation learning comes from the accuracies of architectures selected by the search strategies. Bottom (ours): Disentangling architecture representation learning and architecture search through unsupervised pre-training.

The repository is built upon pytorch_geometric, pybnn, nas_benchmarks, bananas.

1. Requirements

  • NVIDIA GPU, Linux, Python3
pip install -r requirements.txt

2. Experiments on NAS-Bench-101

Dataset preparation on NAS-Bench-101

Install nasbench and download nasbench_only108.tfrecord under ./data folder.

python preprocessing/gen_json.py

Data will be saved in ./data/data.json.

Pretraining

bash models/pretraining_nasbench101.sh

The pretrained model will be saved in ./pretrained/dim-16/.

arch2vec extraction

bash run_scripts/extract_arch2vec.sh

The extracted arch2vec will be saved in ./pretrained/dim-16/.

Alternatively, you can download the pretrained arch2vec on NAS-Bench-101.

Run experiments of RL search on NAS-Bench-101

bash run_scripts/run_reinforce_supervised.sh 
bash run_scripts/run_reinforce_arch2vec.sh 

Search results will be saved in ./saved_logs/rl/dim16

Generate json file:

python plot_scripts/plot_reinforce_search_arch2vec.py 

Run experiments of BO search on NAS-Bench-101

bash run_scripts/run_dngo_supervised.sh 
bash run_scripts/run_dngo_arch2vec.sh 

Search results will be saved in ./saved_logs/bo/dim16.

Generate json file:

python plot_scripts/plot_dngo_search_arch2vec.py

Plot NAS comparison curve on NAS-Bench-101:

python plot_scipts/plot_nasbench101_comparison.py

Plot CDF comparison curve on NAS-Bench-101:

Download the search results from search_logs.

python plot_scripts/plot_cdf.py

3. Experiments on NAS-Bench-201

Dataset preparation

Download the NAS-Bench-201-v1_0-e61699.pth under ./data folder.

python preprocessing/nasbench201_json.py

Data corresponding to the three datasets in NAS-Bench-201 will be saved in folder ./data/ as cifar10_valid_converged.json, cifar100.json, ImageNet16_120.json.

Pretraining

bash models/pretraining_nasbench201.sh

The pretrained model will be saved in ./pretrained/dim-16/.

Note that the pretrained model is shared across the 3 datasets in NAS-Bench-201.

arch2vec extraction

bash run_scripts/extract_arch2vec_nasbench201.sh

The extracted arch2vec will be saved in ./pretrained/dim-16/ as cifar10_valid_converged-arch2vec.pt, cifar100-arch2vec.pt and ImageNet16_120-arch2vec.pt.

Alternatively, you can download the pretrained arch2vec on NAS-Bench-201.

Run experiments of RL search on NAS-Bench-201

CIFAR-10: ./run_scripts/run_reinforce_arch2vec_nasbench201_cifar10_valid.sh
CIFAR-100: ./run_scripts/run_reinforce_arch2vec_nasbench201_cifar100.sh
ImageNet-16-120: ./run_scripts/run_reinforce_arch2vec_nasbench201_ImageNet.sh

Run experiments of BO search on NAS-Bench-201

CIFAR-10: ./run_scripts/run_bo_arch2vec_nasbench201_cifar10_valid.sh
CIFAR-100: ./run_scripts/run_bo_arch2vec_nasbench201_cifar100.sh
ImageNet-16-120: ./run_scripts/run_bo_arch2vec_nasbench201_ImageNet.sh

Summarize search result on NAS-Bench-201

python ./plot_scripts/summarize_nasbench201.py

The corresponding table will be printed to the console.

4. Experiments on DARTS Search Space

CIFAR-10 can be automatically downloaded by torchvision, ImageNet needs to be manually downloaded (preferably to a SSD) from http://image-net.org/download.

Random sampling 600,000 isomorphic graphs in DARTS space

python preprocessing/gen_isomorphism_graphs.py

Data will be saved in ./data/data_darts_counter600000.json.

Alternatively, you can download the extracted data_darts_counter600000.json.

Pretraining

bash models/pretraining_darts.sh

The pretrained model is saved in ./pretrained/dim-16/.

arch2vec extraction

bash run_scripts/extract_arch2vec_darts.sh

The extracted arch2vec will be saved in ./pretrained/dim-16/arch2vec-darts.pt.

Alternatively, you can download the pretrained arch2vec on DARTS search space.

Run experiments of RL search on DARTS search space

bash run_scripts/run_reinforce_arch2vec_darts.sh

logs will be saved in ./darts-rl/.

Final search result will be saved in ./saved_logs/rl/dim16.

Run experiments of BO search on DARTS search space

bash run_scripts/run_bo_arch2vec_darts.sh

logs will be saved in ./darts-bo/ .

Final search result will be saved in ./saved_logs/bo/dim16.

Evaluate the learned cell on DARTS Search Space on CIFAR-10

python darts/cnn/train.py --auxiliary --cutout --arch arch2vec_rl --seed 1
python darts/cnn/train.py --auxiliary --cutout --arch arch2vec_bo --seed 1
  • Expected results (RL): 2.60% test error with 3.3M model params.
  • Expected results (BO): 2.48% test error with 3.6M model params.

Transfer learning on ImageNet

python darts/cnn/train_imagenet.py  --arch arch2vec_rl --seed 1 
python darts/cnn/train_imagenet.py  --arch arch2vec_bo --seed 1
  • Expected results (RL): 25.8% test error with 4.8M model params and 533M mult-adds.
  • Expected results (RL): 25.5% test error with 5.2M model params and 580M mult-adds.

Visualize the learned cell

python darts/cnn/visualize.py arch2vec_rl
python darts/cnn/visualize.py arch2vec_bo

5. Analyzing the results

Visualize a sequence of decoded cells from the latent space

Download pretrained supervised embeddings of nasbench101 and nasbench201.

bash plot_scripts/drawfig5-nas101.sh # visualization on nasbench-101
bash plot_scripts/drawfig5-nas201.sh # visualization on nasbench-201
bash plot_scripts/drawfig5-darts.sh  # visualization on darts

The plots will be saved in ./graphvisualization.

Plot distribution of L2 distance by edit distance

Install nas_benchmarks and download nasbench_full.tfrecord under the same directory.

python plot_scripts/distance_comparison_fig3.py

Latent space 2D visualization

bash plot_scripts/drawfig4.sh

the plots will be saved in ./density.

Predictive performance comparison

Download predicted_accuracy under saved_logs/.

python plot_scripts/pearson_plot_fig2.py

Citation

If you find this useful for your work, please consider citing:

@InProceedings{yan2020arch,
  title = {Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?},
  author = {Yan, Shen and Zheng, Yu and Ao, Wei and Zeng, Xiao and Zhang, Mi},
  booktitle = {NeurIPS},
  year = {2020}
}
GUI for TOAD-GAN, a PCG-ML algorithm for Token-based Super Mario Bros. Levels.

If you are using this code in your own project, please cite our paper: @inproceedings{awiszus2020toadgan, title={TOAD-GAN: Coherent Style Level Gene

Maren A. 13 Dec 14, 2022
PyTorch implementation of MoCo v3 for self-supervised ResNet and ViT.

MoCo v3 for Self-supervised ResNet and ViT Introduction This is a PyTorch implementation of MoCo v3 for self-supervised ResNet and ViT. The original M

Facebook Research 887 Jan 08, 2023
MetaBalance: High-Performance Neural Networks for Class-Imbalanced Data

This repository is the official PyTorch implementation of Meta-Balance. Find the paper on arxiv MetaBalance: High-Performance Neural Networks for Clas

Arpit Bansal 20 Oct 18, 2021
An algorithm study of the 6th iOS 10 set of Boost Camp Web Mobile

알고리즘 스터디 🔥 부스트캠프 웹모바일 6기 iOS 10조의 알고리즘 스터디 입니다. 개인적인 사정 등으로 S034, S055만 참가하였습니다. 스터디 목적 상진: 코테 합격 + 부캠끝나고 아침에 일어나기 위해 필요한 사이클 기완: 꾸준하게 자리에 앉아 공부하기 +

2 Jan 11, 2022
Keeper for Ricochet Protocol, implemented with Apache Airflow

Ricochet Keeper This repository contains Apache Airflow DAGs for executing keeper operations for Ricochet Exchange. Usage You will need to run this us

Ricochet Exchange 5 May 24, 2022
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Website | ICCV paper | arXiv | Twitter This repository contains the official i

Ajay Jain 73 Dec 27, 2022
Official code repository for the publication "Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons"

Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons This repository contains the code to repr

Computational Neuroscience, University of Bern 3 Aug 04, 2022
This repository contains code for the paper "Decoupling Representation and Classifier for Long-Tailed Recognition", published at ICLR 2020

Classifier-Balancing This repository contains code for the paper: Decoupling Representation and Classifier for Long-Tailed Recognition Bingyi Kang, Sa

Facebook Research 820 Dec 26, 2022
A PyTorch Lightning Callback for pushing models to the Hugging Face Hub 🤗⚡️

hf-hub-lightning A callback for pushing lightning models to the Hugging Face Hub. Note: I made this package for myself, mostly...if folks seem to be i

Nathan Raw 27 Dec 14, 2022
Roger Labbe 13k Dec 29, 2022
Spatial Sparse Convolution Library

SpConv: Spatially Sparse Convolution Library PyPI Install Downloads CPU (Linux Only) pip install spconv CUDA 10.2 pip install spconv-cu102 CUDA 11.1 p

Yan Yan 1.2k Jan 07, 2023
Pytorch implementation of PTNet for high-resolution and longitudinal infant MRI synthesis

Pyramid Transformer Net (PTNet) Project | Paper Pytorch implementation of PTNet for high-resolution and longitudinal infant MRI synthesis. PTNet: A Hi

Xuzhe Johnny Zhang 6 Jun 08, 2022
The code for our NeurIPS 2021 paper "Kernelized Heterogeneous Risk Minimization".

Kernelized-HRM Jiashuo Liu, Zheyuan Hu The code for our NeurIPS 2021 paper "Kernelized Heterogeneous Risk Minimization"[1]. This repo contains the cod

Liu Jiashuo 8 Nov 20, 2022
PSGAN running with ncnn⚡妆容迁移/仿妆⚡Imitation Makeup/Makeup Transfer⚡

PSGAN running with ncnn⚡妆容迁移/仿妆⚡Imitation Makeup/Makeup Transfer⚡

WuJinxuan 144 Dec 26, 2022
N-Omniglot is a large neuromorphic few-shot learning dataset

N-Omniglot [Paper] || [Dataset] N-Omniglot is a large neuromorphic few-shot learning dataset. It reconstructs strokes of Omniglot as videos and uses D

11 Dec 05, 2022
It helps user to learn Pick-up lines and share if he has a better one

Pick-up-Lines-Generator(Open Source) It helps user to learn Pick-up lines Share and Add one or many to the DataBase Unique SQLite DataBase AI Undercon

knock_nott 0 May 04, 2022
Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features"

EDM-subgenre-classifier This repository contains the code for "Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Fea

11 Dec 20, 2022
An implementation of Geoffrey Hinton's paper "How to represent part-whole hierarchies in a neural network" in Pytorch.

GLOM An implementation of Geoffrey Hinton's paper "How to represent part-whole hierarchies in a neural network" for MNIST Dataset. To understand this

50 Oct 19, 2022
Augmented CLIP - Training simple models to predict CLIP image embeddings from text embeddings, and vice versa.

Train aug_clip against laion400m-embeddings found here: https://laion.ai/laion-400-open-dataset/ - note that this used the base ViT-B/32 CLIP model. S

Peter Baylies 55 Sep 13, 2022
This repository contains an implementation of ConvMixer for the ICLR 2022 submission "Patches Are All You Need?".

Patches Are All You Need? 🤷 This repository contains an implementation of ConvMixer for the ICLR 2022 submission "Patches Are All You Need?". Code ov

ICLR 2022 Author 934 Dec 30, 2022