Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Overview

RegNet

Designing Network Design Spaces

Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Paper | Official Implementation

RegNet offer a very nice design space for neural network architectures. RegNet design space consists of networks with simple structure which authors call "Regular" Networks (RegNet). Models in RegNet design space have higher concentration of models that perform well and generalise well. RegNet models are very efficient and run upto 5 times faster than EfficientNet models on GPUs.

Also RegNet models have been used as a backbone in Tesla FSD Stack.

Overview Of AnyNet

  • Main goal of the paper is to help in better understanding of network design and discover principles that generalize across settings.
  • Explore structure aspeck of network design and arrive at low dimensional design space consisting of simple regualar networks
  • Network width and depth can be explained by a quantized linear function.

AnyNet Design Space

The basic structure of models in AnyNet design space consists of a simple Stem which is then followed by the network body that does majority of the computation and a final network head that predicts the class scores. The stem and head networks are kept as simple as possible. The network body consists of 4 stages that operate at progressively lower resolutions.

AnyNet

Structure of network body is determined by block width w, network depth d_i, bottleneck ratio b_i and group widths g. Degrees of freedom at stage 'i' are number of blocks d in each stage, block width w and other block parameters such as stride, padding and so on.

Other models are obtained by refining the design space by adding more constraints on the above parameters. Design space is refined keeping the following things in mind :

  • Simplify structure of design space.
  • Improve the interpretability of design space.
  • Maintain Design space complexity.
  • Maintain model diversity in design space.

AnyNetX

XBlock

  • Uses XBlocks within each block of the network
  • Degrees of freedom in AnyNetX is 16
  • Each network has 4 stages
  • Each stage has 4 parameters (network depth di, block width wi, bottleneck ratio bi, group width gi)
  • bi ∈ {1,2,4}
  • gi ∈ {1,2,3,...,32}
  • wi <= 1024
  • di <= 16

AnyNetX(A)

AnyNetX(A) is same as the above AnyNetX

AnyNetX(B)

In this design space,

  • bottleneck ratio bi is fixed for all stages.
  • performance of models in AnyNetX(B) space is almost equal to AnyNetX(A) in average and best case senarios
  • bi <= 2 seemes to work best.

AnyNetX(C)

In this design space,

  • Shared group width gi for all stages.
  • AnyNetX(C) has 6 fewer degrees of freedom compared to AnyNetX(A)
  • gi > 1 seems to work best

AnyNetX(D)

In AnyNetX(D) design space, authors observed that good networks have increasing stage widths w(i+1) > wi

AnyNetX(E)

In AnyNetX(E) design space, it was observed that as stage widths wi increases, depth di likewise tend to increase except for the last stage.

RegNet

Please refer to Section 3.3 in paper.

Training

Import any of the following variants of RegNet using

from regnet import regnetx_002 as RegNet002
from regnet import Xblock, Yblock # required if you want to use YBlock instead of Xblock. Refer to paper for more details on YBlock

RegNet variants available are:

  • regnetx_002
  • regnetx_004
  • regnetx_006
  • regnetx_008
  • regnetx_016
  • regnetx_032
  • regnetx_040
  • regnetx_064
  • regnetx_080
  • regnetx_120
  • regnetx_160
  • regnetx_320

Import TrainingConfig and Trainer Classes from regnet and use them to train the model as follows

from regnet import TrainingConfig, Trainer

model = RegNet002(block=Xblock, num_classes=10)

training_config = TrainingConfig(max_epochs=10, batch_size=128, learning_rate=3e-4, weight_decay=5e-4, ckpt_path="./regnet.pt")
trainer = Trainer(model = model, train_dataset=train_dataset, test_dataset=test_dataset, config=training_config)
trainer.train()

Note : you need not use TrainingConfig and Trainer classes if you want to write your own training loops. Just importing the respective models would suffice.

TODO

  • Test if model trains when using YBlocks
  • Implement model checkpointing for every 'x' epochs

References

[1] https://github.com/signatrix/regnet

[2] https://github.com/d-li14/regnet.pytorch

@InProceedings{Radosavovic2020,
  title = {Designing Network Design Spaces},
  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Doll{\'a}r},
  booktitle = {CVPR},
  year = {2020}
}

LICENSE

MIT

Owner
Vishal R
Computer Science Student at PES University.
Vishal R
NEO: Non Equilibrium Sampling on the orbit of a deterministic transform

NEO: Non Equilibrium Sampling on the orbit of a deterministic transform Description of the code This repo describes the NEO estimator described in the

0 Dec 01, 2021
The source code of CVPR 2019 paper "Deep Exemplar-based Video Colorization".

Deep Exemplar-based Video Colorization (Pytorch Implementation) Paper | Pretrained Model | Youtube video 🔥 | Colab demo Deep Exemplar-based Video Col

Bo Zhang 253 Dec 27, 2022
Little Ball of Fur - A graph sampling extension library for NetworKit and NetworkX (CIKM 2020)

Little Ball of Fur is a graph sampling extension library for Python. Please look at the Documentation, relevant Paper, Promo video and External Resour

Benedek Rozemberczki 619 Dec 14, 2022
An open-source, low-cost, image-based weed detection device for fallow scenarios.

Welcome to the OpenWeedLocator (OWL) project, an opensource hardware and software green-on-brown weed detector that uses entirely off-the-shelf compon

Guy Coleman 145 Jan 05, 2023
NLU Dataset Diagnostics

NLU Dataset Diagnostics This repository contains data and scripts to reproduce the results from our paper: Aarne Talman, Marianna Apidianaki, Stergios

Language Technology at the University of Helsinki 1 Jul 20, 2022
Sparse Physics-based and Interpretable Neural Networks

Sparse Physics-based and Interpretable Neural Networks for PDEs This repository contains the code and manuscript for research done on Sparse Physics-b

28 Jan 03, 2023
Pytorch implementation of "Neural Wireframe Renderer: Learning Wireframe to Image Translations"

Neural Wireframe Renderer: Learning Wireframe to Image Translations Pytorch implementation of ideas from the paper Neural Wireframe Renderer: Learning

Yuan Xue 7 Nov 14, 2022
PyTorch Implementation of Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation

StyleSpeech - PyTorch Implementation PyTorch Implementation of Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation. Status (2021.06.13

Keon Lee 140 Dec 21, 2022
A PaddlePaddle implementation of Time Interval Aware Self-Attentive Sequential Recommendation.

TiSASRec.paddle A PaddlePaddle implementation of Time Interval Aware Self-Attentive Sequential Recommendation. Introduction 论文:Time Interval Aware Sel

Paddorch 2 Nov 28, 2021
End-to-End Object Detection with Fully Convolutional Network

This project provides an implementation for "End-to-End Object Detection with Fully Convolutional Network" on PyTorch.

472 Dec 22, 2022
Implementation of Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

acLSTM_motion This folder contains an implementation of acRNN for the CMU motion database written in Pytorch. See the following links for more backgro

Yi_Zhou 61 Sep 07, 2022
Si Adek Keras is software VR dangerous object detection.

Si Adek Python Keras Sistem Informasi Deteksi Benda Berbahaya Keras Python. Version 1.0 Developed by Ananda Rauf Maududi. Developed date: 24 November

Ananda Rauf 1 Dec 21, 2021
Automatically align face images 🙃→🙂. Can also do windowing and warping.

Automatic Face Alignment (AFA) Carl M. Gaspar & Oliver G.B. Garrod You have lots of photos of faces like this: But you want to line up all of the face

Carl Michael Gaspar 15 Dec 12, 2022
A curated list of awesome papers for Semantic Retrieval (TOIS Accepted: Semantic Models for the First-stage Retrieval: A Comprehensive Review).

A curated list of awesome papers for Semantic Retrieval (TOIS Accepted: Semantic Models for the First-stage Retrieval: A Comprehensive Review).

Yinqiong Cai 189 Dec 28, 2022
Real-time 3D multi-person detection made easy with OpenPose and the ZED

OpenPose ZED This sample show how to simply use the ZED with OpenPose, the deep learning framework that detects the skeleton from a single 2D image. T

blanktec 5 Nov 06, 2020
This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Anirudh S Chakravarthy 6 May 03, 2022
2021-MICCAI-Progressively Normalized Self-Attention Network for Video Polyp Segmentation

2021-MICCAI-Progressively Normalized Self-Attention Network for Video Polyp Segmentation Authors: Ge-Peng Ji*, Yu-Cheng Chou*, Deng-Ping Fan, Geng Che

Ge-Peng Ji (Daniel) 85 Dec 30, 2022
QuakeLabeler is a Python package to create and manage your seismic training data, processes, and visualization in a single place — so you can focus on building the next big thing.

QuakeLabeler Quake Labeler was born from the need for seismologists and developers who are not AI specialists to easily, quickly, and independently bu

Hao Mai 15 Nov 04, 2022
PyTorch implementation of some learning rate schedulers for deep learning researcher.

pytorch-lr-scheduler PyTorch implementation of some learning rate schedulers for deep learning researcher. Usage WarmupReduceLROnPlateauScheduler Visu

Soohwan Kim 59 Dec 08, 2022
Segmentation models with pretrained backbones. PyTorch.

Python library with Neural Networks for Image Segmentation based on PyTorch. The main features of this library are: High level API (just two lines to

Pavel Yakubovskiy 6.6k Jan 06, 2023