Optimizer

pytorch Optimizer ： Manage and update the values of learnable parameters in the model , Make the model output closer to the real label

Basic attributes ：
defaults: Optimizer parameters
state: Parameter cache , Such as momentum The cache of
param_groups: Managed parameter groups
_step_count: Record the number of updates , Use... In learning rate adjustment
operation ：
zero_grad(): Clear the gradient of managed parameters
step(): Perform one step update
add_param_group(): Add parameter group
state_dict()： Get the optimizer's current state information dictionary
load_state_dict()： Load state information dictionary

1. nn.optim.SGD
   main parameter ：
   params: Managed parameter groups
   lr： Initial learning rate
   momentum: Momentum coefficient
   weight_decay:L2 Regularization coefficient
   nesterov: Whether to adopt NAG, Nesterov gradient optimization method
2. nn.optim.Adagrad
   Adaptive learning rate gradient descent method
3. nn.optim.RMSprop
   Adagrad Improvement
4. nn.optim.Adam
   RMSprop combination Momentum
5. nn.optim.AdamW
   Weighted adam