Learning Versatile Neural Architectures by Propagating Network Codes

Last update: Dec 06, 2022

Related tags

Overview

Learning Versatile Neural Architectures by Propagating Network Codes

Mingyu Ding, Yuqi Huo, Haoyu Lu, Linjie Yang, Zhe Wang, Zhiwu Lu, Jingdong Wang, Ping Luo

Introduction

This work includes:
(1) NAS-Bench-MR, a NAS benchmark built on four challenging datasets under practical training settings for learning task-transferable architectures.
(2) An efficient predictor-based algorithm Network Coding Propagation (NCP), which back-propagates the gradients of neural predictors to directly update architecture codes along desired gradient directions for various objectives.

This framework is implemented and tested with Ubuntu/Mac OS, CUDA 9.0/10.0, Python 3, Pytorch 1.3-1.6, NVIDIA Tesla V100/CPU.

Dataset

We build our benchmark on four computer vision tasks, i.e., image classification (ImageNet), semantic segmentation (CityScapes), 3D detection (KITTI), and video recognition (HMDB51). Totally 9 different settings are included, as shown in the data/*/trainval.pkl folders.

Note that each .pkl file contains more than 2500 architectures, and their corresponding evaluation results under multiple metrics. The original training logs and checkpoints (including model weights and optimizer data) will be uploaded to Google drive (more than 4T). We will share the download link once the upload is complete.

Quick start

First, train the predictor

python3 tools/train_predictor.py  # --cfg configs/seg.yaml

Then, edit architecture based on desired gradients

python3 tools/ncp.py  # --cfg configs/seg.yaml

Examples

An example in NAS-Bench-MR (Seg):

{'mIoU': 70.57,
 'mAcc': 80.07,
 'aAcc': 95.29,
 'input_channel': [16, 64],
 # [num_branches, [num_convs], [num_channels]]
 'network_setting': [[1, [3], [128]],
  [2, [3, 3], [32, 48]],
  [2, [3, 3], [32, 48]],
  [2, [3, 3], [32, 48]],
  [3, [2, 3, 2], [16, 32, 16]],
  [3, [2, 3, 2], [16, 32, 16]],
  [4, [2, 4, 1, 1], [96, 112, 48, 80]]],
 'last_channel': 112,
 # [num_branches, num_block1, num_convs1, num_channels1, ..., num_block4, num_convs4, num_channels4, last_channel]
 'embedding': [16, 64, 1, 3, 128, 3, 3, 3, 32, 48, 2, 2, 3, 2, 16, 32, 16, 1, 2, 4, 1, 1, 96, 112, 48, 80]
}

Load Datasets:

import pickle
exps = pickle.load(open('data/seg/trainval.pkl', 'rb'))
# Then process each item in exps

Load Model / Get Params and Flops (based on the thop library):

import torch
from thop import profile
from models.supernet import MultiResolutionNet

# Get model using input_channel & network_setting & last_channel
model = MultiResolutionNet(input_channel=[16, 64],
                           network_setting=[[1, [3], [128]],
                            [2, [3, 3], [32, 48]],
                            [2, [3, 3], [32, 48]],
                            [2, [3, 3], [32, 48]],
                            [3, [2, 3, 2], [16, 32, 16]],
                            [3, [2, 3, 2], [16, 32, 16]],
                            [4, [2, 4, 1, 1], [96, 112, 48, 80]]],
                          last_channel=112)

# Get Flops and Parameters
input = torch.randn(1, 3, 224, 224)
macs, params = profile(model, inputs=(input, ))

Data Format

Each code in data/search_list.txt denotes an architecture. It can be load in our supernet as follows:

Code2Setting

params = '96_128-1_1_1_48-1_2_1_1_128_8-1_3_1_1_1_128_128_120-4_4_4_4_4_4_128_128_128_128-64'
embedding = [int(item) for item in params.replace('-', '_').split('_')]

embedding = [ 96, 128,   1,   1,  48,   1,   1,   1, 128,   8,   1,   1,
           1,   1, 128, 128, 120,   4,   4,   4,   4,   4, 128, 128,
         128, 128, 64]
input_channels = embedding[0:2]
block_1 = embedding[2:3] + [1] + embedding[3:5]
block_2 = embedding[5:6] + [2] + embedding[6:10]
block_3 = embedding[10:11] + [3] + embedding[11:17]
block_4 = embedding[17:18] + [4] + embedding[18:26]
last_channels = embedding[26:27]
network_setting = []
for item in [block_1, block_2, block_3, block_4]:
    for _ in range(item[0]):
        network_setting.append([item[1], item[2:-int(len(item) / 2 - 1)], item[-int(len(item) / 2 - 1):]])

# network_setting = [[1, [1], [48]], 
#  [2, [1, 1], [128, 8]],
#  [3, [1, 1, 1], [128, 128, 120]], 
#  [4, [4, 4, 4, 4], [128, 128, 128, 128]], 
#  [4, [4, 4, 4, 4], [128, 128, 128, 128]], 
#  [4, [4, 4, 4, 4], [128, 128, 128, 128]], 
#  [4, [4, 4, 4, 4], [128, 128, 128, 128]]]
# input_channels = [96, 128]
# last_channels = [64]

Setting2Code

input_channels = [str(item) for item in input_channels]
block_1 = [str(item) for item in block_1]
block_2 = [str(item) for item in block_2]
block_3 = [str(item) for item in block_3]
block_4 = [str(item) for item in block_4]
last_channels = [str(item) for item in last_channels]

params = [input_channels, block_1, block_2, block_3, block_4, last_channels]
params = ['_'.join(item) for item in params]
params = '-'.join(params)
# params
# 96_128-1_1_1_48-1_2_1_1_128_8-1_3_1_1_1_128_128_120-4_4_4_4_4_4_128_128_128_128-64'

License

For academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact the author.

Learning Versatile Neural Architectures by Propagating Network Codes

Related tags

Overview

Learning Versatile Neural Architectures by Propagating Network Codes

Introduction

Dataset

Quick start

First, train the predictor

Then, edit architecture based on desired gradients

Examples

Data Format

License

Owner

Mingyu Ding

B-cos Networks: Attention is All we Need for Interpretability

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

A deep learning framework for historical document image analysis

[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

BRNet - code for Automated assessment of BI-RADS categories for ultrasound images using multi-scale neural networks with an order-constrained loss function

Optimal space decomposition based-product quantization for approximate nearest neighbor search

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.

Kaggle: Cell Instance Segmentation

Human Pose Detection on EdgeTPU

Basics of 2D and 3D Human Pose Estimation.

Disagreement-Regularized Imitation Learning

Baseline powergrid model for NY

A face dataset generator with out-of-focus blur detection and dynamic interval adjustment.

This is the implementation of "SELF SUPERVISED REPRESENTATION LEARNING WITH DEEP CLUSTERING FOR ACOUSTIC UNIT DISCOVERY FROM RAW SPEECH" submitted to ICASSP 2022

Reinforcement learning models in ViZDoom environment

Digitalizing-Prescription-Image - PIRDS - Prescription Image Recognition and Digitalizing System is a OCR make with Tensorflow

CarND-LaneLines-P1 - Lane Finding Project for Self-Driving Car ND

Semantic Scholar's Author Disambiguation Algorithm & Evaluation Suite

Learning Versatile Neural Architectures by Propagating Network Codes

Related tags

Overview

Learning Versatile Neural Architectures by Propagating Network Codes

Introduction

Dataset

Quick start

First, train the predictor

Then, edit architecture based on desired gradients

Examples

Data Format

License

Owner

Mingyu Ding

B-cos Networks: Attention is All we Need for Interpretability

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

A deep learning framework for historical document image analysis

[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

BRNet - code for Automated assessment of BI-RADS categories for ultrasound images using multi-scale neural networks with an order-constrained loss function

Optimal space decomposition based-product quantization for approximate nearest neighbor search

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

​TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.

Kaggle: Cell Instance Segmentation

Human Pose Detection on EdgeTPU

Basics of 2D and 3D Human Pose Estimation.

Disagreement-Regularized Imitation Learning

Baseline powergrid model for NY

A face dataset generator with out-of-focus blur detection and dynamic interval adjustment.

This is the implementation of "SELF SUPERVISED REPRESENTATION LEARNING WITH DEEP CLUSTERING FOR ACOUSTIC UNIT DISCOVERY FROM RAW SPEECH" submitted to ICASSP 2022

Reinforcement learning models in ViZDoom environment

Digitalizing-Prescription-Image - PIRDS - Prescription Image Recognition and Digitalizing System is a OCR make with Tensorflow

CarND-LaneLines-P1 - Lane Finding Project for Self-Driving Car ND

Semantic Scholar's Author Disambiguation Algorithm & Evaluation Suite

TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.