Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Code for “Efficient Sharpness-aware Minimization for Improved Training of Neural Networks”

Requisite

This code is implemented in PyTorch, and we have tested the code under the following environment settings:

python = 3.8.8
torch = 1.8.0
torchvision = 0.9.0

What is in this repository

Codes for our ESAM on CIFAR10/CIFAR100 datasets.

How to use it

from utils.layer_dp_sam import ESAM
base_optimizer = torch.optim.SGD(model.parameters(),lr=args.learning_rate,momentum=0.9,weight_decay=args.weight_decay)
optimizer = ESAM(paras, base_optimizer, rho=args.rho, weight_dropout=args.weight_dropout,adaptive=args.isASAM,nograd_cutoff=args.nograd_cutoff,opt_dropout = args.opt_dropout,temperature=args.temperature)

--beta the SWP hyperparameter

--gamma the SDS hyperparameter

During training loss_fct should have reduction="none", to return instance-wise losses. defined_backward is the function used for DDP and mixed precision backward

loss_fct = torch.nn.CrossEntropyLoss(reduction="none")
def defined_backward():
    if args.fp16:
    with amp.scale_loss(loss, optimizer0) as scaled_loss:
        scaled_loss.backward()
    else:
        loss.backward()

paras = [inputs,targets,loss_fct,model,defined_backward]
optimizer.paras = paras
optimizer.step()
predictions_logits,loss = optimizer.returnthings

Example

bash run.sh

Reference Code

[1] SAM

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Related tags

Overview

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Requisite

What is in this repository

How to use it

Example

Reference Code

Owner

Angusdu

Source code of our work: "Benchmarking Deep Models for Salient Object Detection"

MultiLexNorm 2021 competition system from ÚFAL

An NLP library with Awesome pre-trained Transformer models and easy-to-use interface, supporting wide-range of NLP tasks from research to industrial applications.

A Python package for faster, safer, and simpler ML processes

We present a regularized self-labeling approach to improve the generalization and robustness properties of fine-tuning.

Official PyTorch implementation of RobustNet (CVPR 2021 Oral)

This is the code for the paper "Contrastive Clustering" (AAAI 2021)

Pytorch Geometric Tutorials

Code from Daniel Lemire, A Better Alternative to Piecewise Linear Time Series Segmentation

learned_optimization: Training and evaluating learned optimizers in JAX

High accurate tool for automatic faces detection with landmarks

Library for machine learning stacking generalization.

Tool for installing and updating MiSTer cores and other files

Official Pytorch implementation of C3-GAN

Cweqgen - The CW Equation Generator

[Arxiv preprint] Causality-inspired Single-source Domain Generalization for Medical Image Segmentation (code&data-processing pipeline)

Manipulation OpenAI Gym environments to simulate robots at the STARS lab

Official Implementation of SWAGAN: A Style-based Wavelet-driven Generative Model

Automatic self-diagnosis program (python required)Automatic self-diagnosis program (python required)

Caffe: a fast open framework for deep learning.