Deep learning - Super parameter setting

One . Over fitting
1. Definition ： Given a hypothetical space H, A hypothesis h Belong to H, If there are other assumptions h’ Belong to H, So in the training example h The error rate of h’ Small , But in the whole instance distribution h’ Than h Low error rate , So let's say suppose h Overfitting training data .
2. Popular explanation
3. Common causes
It is mainly over learning and unbalanced sample characteristics , If the segment , It can also include （ Not all the reasons ）：
（1） Wrong modeling sample selection , Sample label error, etc , Causes the selected sample data to be insufficient to represent the intended classification rule
（2） Excessive sample noise interference , Make the machine learn the noise , Also considered a feature , Thus disturbing the preset classification rules
（3） The hypothetical model cannot reasonably exist , Or the conditions under which the hypothesis is true are not true （4） Too many parameters , Excessive model complexity
（5） about tree-based Model , If we compare its depth with split There are no reasonable restrictions , It is possible to make the node contain only simple event data (event) Or non-event data (no event), Make it a perfect match though （ fitting ） Training data , But it can't adapt to other data sets
（6） For the neural network model ：1). Too many iterations of weight learning (Overtraining),2).BP The algorithm may make the weight converge to the decision surface which is too complex .
4. resolvent
-> Model ： neural network ： Add dropout,batch normalization Tree based model ： Limit depth , Add regularization items and set early termination conditions .
-> Data on ： Increase the data set and enhance the data set （augmentation).
Two 、 Regularization
Preliminary knowledge （ Gradient descent method ）：https://zhuanlan.zhihu.com/p/113714840
1. The purpose of regularization ： A weight accumulation term added for the generalization of the model .