一、模型model

• 一个函数function的集合：

  • 其中w_i代表权重weight,b代表偏置值bias
  • 𝑥_𝑖Different properties can be taken,如: 𝑥_𝑐𝑝, 𝑥_ℎ𝑝, 𝑥_𝑤,𝑥_ℎ…

  $$y=b+\sum w_i{x}_i$$

• 我们将𝑥_𝑐𝑝Take it out as an unknown quantity,to find an optimal linear modelLinear model：
  $$y=b+w\bullet Xcp$$

二、better functionfunction

• 损失函数Loss function 𝐿：

  • L的输入Input是一个函数 f ,输出outputis a specific value,And this value is the function used to evaluate the input f 到底有多坏

  • ${\hat{y}}^n$代表真实值,而$f(x_{cp}^n)$代表预测值,$L(f)$represents the total error between the true value and the predicted value
    $$L(f)=\sum _{n=1}^{10}({\hat{y}}^n-f(x_{cp}^n){)}^2$$

  • 将函数 f 用w,b替换,则可以写成下面这样
    $$L(w,b)=\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$$

  • 当 L 越小时,则说明该函数 f 越好,That is, the model is better.Each point in the graph below represents a function f

    

三、best functionfunction

• 梯度下降Gradient Descent：It is the process of finding the best function

• $f^{ } $represents\; the\; best\; functionfunction,$w^{*},b{ }：$represents the best weightweight和偏置值bias

  $f^{\ast }=arg\underset{f}{\min }L(f)$

  $w^{\ast },b^{\ast }=arg\underset{w,b}{\min }L(w,b)$

  ​ $=arg\underset{w,b}{\min }{\sum }_{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$

3.1 一维函数

下图代表Lossfunction for gradient descent（Gradient Descent）的过程,首先随机选择一个$w^0$.at that pointw求微分,如果为负数,Then we increase$w^0$的值;如果为正数,Then we reduce$w^0$的值.

• $w^{\ast }=arg\underset{w}{\min }L(w)$
• $w^0=-\eta \frac{dL}{dw}{|}_w$,其中 η 代表学习率：Learning rate,means the step size for each move（step）
• $w^1\leftarrow w^0-\eta \frac{dL}{dw}{|}_{w=w^0}$,$w1$represents the initial point$w^0$The next point to move,Iterate like this（Iteration）下去,Eventually our local optimum will be found：Local optimal solution

3.2 二维函数

• for two-dimensional functions$Loss $ $L(w,b)$Find gradient descent：${\left[\begin{array}{c}\frac{\partial L}{\partial w}\\ \frac{\partial L}{\partial b}\end{array}\right]}_{gradient}$
• $w^{\ast },b^{\ast }=arg\underset{w,b}{\min }L(w,b)$
• 随机初始化$w^0,b^0$,然后计算$\frac{\partial L}{\partial w}{|}_{w=w^0,b=b^0}$和$\frac{\partial L}{\partial b}{|}_{w=w^0,b=b^0}$：
  • $w^1\leftarrow w^0-\eta \frac{\partial L}{\partial w}{|}_{w=w^0,b=b^0}$
  • $b^1\leftarrow b^0-\eta \frac{\partial L}{\partial b}{|}_{w=w^0,b=b^0}$

3.3 局部最优解和全局最优解

• 公式化（Formulation）$\frac{\partial L}{\partial w}$和$
  \frac{\partial L}{\partial b}$：

  • $L(w,b)={\sum }_{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$
  • $\frac{\partial L}{\partial w}=2{\sum }_{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n))(-x_{cp}^n)$
  • $\frac{\partial L}{\partial b}=2{\sum }_{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n))$

• 在非线性系统中,There may be multiple local optimal solutions：

  

3.4 模型的泛化（Generalization）能力

• 将根据lossThe best model found by the function is taken out,Calculate it separately on the training set（Training Data）和测试集（Testing Data）mean squared error on （Average Error）,Of course, we only care about how well the model performs on the test set.

  • $y=b+w\bullet x_{cp}$ Average Error=35.0

• Because the mean square error of the original model is still relatively large,为了做得更好,Let's increase the complexity of the model.比如,引入二次项(x_cp)²

  • $y=b+w1\bullet x_{cp}+w2\bullet (x_{cp)}2$ Average Error = 18.4

• Continue to increase the complexity of the model,Introduce three terms(x_cp)³

  • $y=b+w1\bullet x_{cp}+w2\bullet (x_{cp})2+w3\bullet (x_{cp})3$ Average Error = 18.1

• Continue to increase the complexity of the model,Introduce three terms(x_cp)⁴,At this point, the mean squared error of the model on the training set becomes smaller,But the test set has become larger,This phenomenon is called overfitting of the model（Over-fitting）

  • $y = b + w1∙x_{cp} + w2∙(x_{cp})2+ w3∙(x_{cp})3+ w4∙(x_{cp})4 $ Average Error = 28.8

3.5 hidden factor（hidden factors）

• When we don't just think about Pokémoncp值,Taking the species of Pokémon into account,The mean squared error on the test set is reduced to 14.3

• As we move on to consider other factors,Such as the height of each PokémonHeight,体重weight,经验值HP.The model becomes more complex at this point,Let's see how it performs on the test set,Very unfortunately the model overfits again.

3.5 正则化（Regularization）

• 为了解决过拟合的问题,We need to redesign the loss function L,The original loss function only calculated the variance,It does not take into account the influence of the input containing noise on the model.因此我们在 L Add an item after: $\lambda \sum (w_i{)}^2$ ,This improves the generalization ability of the model,Make the model smoother,Reduce the sensitivity of the model to the input（Sensitive）

  • Redesigned loss function L ：$L(f)=\sum _n({\hat{y}}^n-(b+\sum w_i{x}_i){)}^2+\lambda \sum (w_i{)}^2$

• Obviously according to the following experiment,We got better performance,$当\lambda =100时,TestError=11.1$