scheduler

scheduler: An optimizer must adjust its learning rate

class _LRScheduler(object):
	def __init__(self, optimizer, last_epoch=-1):
		pass
	def get_lr(self):
		return [base_lr * self.gamma ** (self.last_epoch//self.step_size) for base_lr in self.base_lrs]
	def step(self):
		if epoch is None:
			epoch = self.last_epoch + 1
		self.last_epoch = epoch
		for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):
			param_group['lr'] = lr

optimizer： Associated optimizer
last_epoch: Record epoch Count
base_lrs: Record the initial learning rate
The main method ：
step()： Update next epoch Learning rate of
get_lr(): Calculate next epoch Learning rate of

StepLR

lr_scheduler.StepLR(optimizer, step_size, gamma=0.1,last_epoch=-1)

function ： Adjust the learning rate at equal intervals
main parameter ：
step_size: Adjust the number of intervals
gamma: Adjustment factor
arrange mode ： lr = lr * gamma

MultiStepLR

lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)

function ： Adjust the learning rate at a given interval
main parameter :
milestones： Set the number of adjustment times ,milestones = [50, 125, 160]
gamma: Adjustment factor
arrange mode ： lr = lr *gamma

ExponentialLR

lr_scheduler.ExponentialLR(optimizer, gamma, last-epoch=-1)

function ： Adjust the learning rate by exponential decay
main parameter ：
gamma: The bottom of the index

CosineAnnealingLR

lr_scheduler.CosineAnnealingLR(optimizer, T_max, eta_min=0, last_epoch=-1)

function ： Cosine period adjusted learning rate
main parameter ：
T_max: Descent cycle
eta_min: Lower limit of learning rate
arrange mode ：
$l{r}_t=l{r}_{min}+\frac{1}{2}(l{r}_{max}-l{r}_{min})(1+cos(\frac{T_{cur}}{T_{max}}\pi ))$

ReduceLRonPlateau

lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel',cooldown=0, min_lr=0, eps=1e-08)

function ： Monitoring indicators , When the index no longer changes, adjust
main parameter ：
mode: min/max Two modes
factor: Adjustment factor
patience: “ Patience, ”, Accept several times without change
cooldown: “ Cooling time ”, Stop monitoring for a while
verbose: Whether to print the log
min_lr: Lower limit of learning rate
eps: Minimum attenuation of learning rate

Reference resources ：
https://zhuanlan.zhihu.com/p/146865009