Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Last update: Sep 16, 2022

Related tags

Deep Learning AngularGrad

Overview

AngularGrad Optimizer

This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks in PyTorch.

AngularGrad reduces the zig-zag effect in the optimization trajectory. Fluctuations are significantly smoothed, tracing a more direct path towards the minimum of the cost function.

If you have questions or suggestions, please feel free to open an issue. Please cite as:

@article{roy2021angulargrad,
  title={AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks},
  author={S.K. Roy, M.E. Paoletti, J.M. Haut, S.R. Dubey, P. Kar, A. Plaza and B.B. Chaudhuri},
  journal={arXiv preprint arXiv:2105.10190},
  year={2021}
}

In PyTorch:

optimizer = cosangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
<or>
optimizer = tanangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
Example:
optimizer = cosangulargrad(net.parameters())

Experiments

Experiments in the paper:

Analitycal

cd analitycal/
python main.py

CIFAR-10/100

cd cifar/
python main.py --dataset <cifar10/cifar100> --model <r18/r34/.../vgg16/d121> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py --dataset cifar10 --model r50 --alg cosangulargrad --lr 1e-3

Mini-ImageNet:

cd mini-imagenet/
wget URL dataset
python main.py DATADIR --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./split_mini/ --alg cosangulargrad --model r50 --lr 1e-3

Fine-Grained:

cd fine-grained/
wget URL datasets
python main.py DATADIR --dataset <dsetname> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./data/Car196/ --dataset cars --alg adam --lr 1e-3

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Related tags

Overview

AngularGrad Optimizer

Experiments

Owner

mario

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers

Identifying a Training-Set Attack’s Target Using Renormalized Influence Estimation

PyTorch implementation for our paper "Deep Facial Synthesis: A New Challenge"

Asynchronous Advantage Actor-Critic in PyTorch

Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

I will implement Fastai in each projects present in this repository.

Efficient Two-Step Networks for Temporal Action Segmentation (Neurocomputing 2021)

Deformable DETR is an efficient and fast-converging end-to-end object detector.

MANO hand model porting for the GraspIt simulator

Style transfer, deep learning, feature transform

An experiment to bait a generalized frontrunning MEV bot

KE-Dialogue: Injecting knowledge graph into a fully end-to-end dialogue system.

An inofficial PyTorch implementation of PREDATOR based on KPConv.

Learning Chinese Character style with conditional GAN

Code for the paper "Controllable Video Captioning with an Exemplar Sentence"

Stock-Prediction - prediction of stock market movements using sentiment analysis and deep learning.

This repository comes with the paper "On the Robustness of Counterfactual Explanations to Adverse Perturbations"

This is the repository for the paper "Have I done enough planning or should I plan more?"

Convnext-tf - Unofficial tensorflow keras implementation of ConvNeXt