optimization

The goal of optimization ： Find a series of weights that minimize the value of the loss function w

Random initialization

import torch
from torch.utils.data import DataLoader
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import numpy as np

class_num = 10
batch_size = 4
label = torch.LongTensor(batch_size, 1).random_() % class_num
print(label)
#  hold Tensor convert to one_hot   scatter_(1,label,1)  Indicating taking 1 according to label Matrix to right zeros The matrix is filled with columns 
ont_hot = torch.zeros(batch_size,class_num).scatter_(1,label,1)
print('---')
print(ont_hot)

Gradient descent method

The code implementation is as follows ：

def gradient_descent(f, init_X,lr=0.01,step_num=100):
    x = init_X
    for i in range(step_num):
        grad = numberical_gradient(f,x)
        x =-= lr*grad
    return x

Parameters f It's a function to optimize ,init_x Is the initial value ,lr It means the learning rate ,step_num Represents the number of repetitions of the gradient descent method

Add concept ：Np.nditer

The code is as follows ：

import numpy as np

arr1 = np.arange(0,30,5).reshape(2,3)
it = np.nditer(arr1, flags=['multi_index'], op_flags=['readwrite'])
while not it.finished:
    print(it.multi_index)
    it.iternext()

Running results ：

op_flags=['readwrite'] Means not only to a To read , You can also write

print(it.multi_index) Represents the index of the output element

it.iternext() Indicates entering the next iteration