1 Learning rate warm-up introduction

• In the mini-batch gradient descent method, if the batch is relatively large, a relatively large learning rate is usually required
• But in the initial training, since the parameters are randomly initialized, the gradient at this time is often very large.
  • If the learning rate is also large at this time, the training will become unstable
• ——>In order to improve the stability of training, we use a smaller learning rate in the first few iterations, wait for the gradient to drop to a certain extent, and then restore to the initial learning rate
  • This method is called learning rate warmup
  • When the warm-up process is over, choose a learning rate decay method to reduce the learning rate

2 Gradual warmup gradual warmup

A common method is to warm up gradually.Assuming that the number of iterations of warm-up is T' and the initial learning rate is α0, then in the process of warm-up, the learning rate of each update is