Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Last update: Sep 16, 2022

Related tags

Deep Learning AngularGrad

Overview

AngularGrad Optimizer

This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks in PyTorch.

AngularGrad reduces the zig-zag effect in the optimization trajectory. Fluctuations are significantly smoothed, tracing a more direct path towards the minimum of the cost function.

If you have questions or suggestions, please feel free to open an issue. Please cite as:

@article{roy2021angulargrad,
  title={AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks},
  author={S.K. Roy, M.E. Paoletti, J.M. Haut, S.R. Dubey, P. Kar, A. Plaza and B.B. Chaudhuri},
  journal={arXiv preprint arXiv:2105.10190},
  year={2021}
}

In PyTorch:

optimizer = cosangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
<or>
optimizer = tanangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
Example:
optimizer = cosangulargrad(net.parameters())

Experiments

Experiments in the paper:

Analitycal

cd analitycal/
python main.py

CIFAR-10/100

cd cifar/
python main.py --dataset <cifar10/cifar100> --model <r18/r34/.../vgg16/d121> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py --dataset cifar10 --model r50 --alg cosangulargrad --lr 1e-3

Mini-ImageNet:

cd mini-imagenet/
wget URL dataset
python main.py DATADIR --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./split_mini/ --alg cosangulargrad --model r50 --lr 1e-3

Fine-Grained:

cd fine-grained/
wget URL datasets
python main.py DATADIR --dataset <dsetname> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./data/Car196/ --dataset cars --alg adam --lr 1e-3

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Related tags

Overview

AngularGrad Optimizer

Experiments

Owner

mario

GANSketchingJittor - Implementation of Sketch Your Own GAN in Jittor

Picasso: A CUDA-based Library for Deep Learning over 3D Meshes

AoT is a system for automatically generating off-target test harness by using build information.

Towards uncontrained hand-object reconstruction from RGB videos

Does Pretraining for Summarization Reuqire Knowledge Transfer?

SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation

“Data Augmentation for Cross-Domain Named Entity Recognition” (EMNLP 2021)

Code for the paper "Can Active Learning Preemptively Mitigate Fairness Issues?" presented at RAI 2021.

YoHa - A practical hand tracking engine.

A boosting-based Multiple Instance Learning (MIL) package that includes MIL-Boost and MCIL-Boost

Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

Predicting Student Attentiveness using OpenCV

Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Official code for the ICCV 2021 paper "DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders"

Dynamic View Synthesis from Dynamic Monocular Video

Proto-RL: Reinforcement Learning with Prototypical Representations

A simple program for training and testing vit

HistoKT: Cross Knowledge Transfer in Computational Pathology

This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.