Deeply learn the skills of parameter adjustment

How to adjust parameters ：

1. batchsize Be suitable

2. epoch Be suitable , Observe the convergence , Prevent over fitting

3. Whether to add batch nomal

4. dropout If you need

5. Activate function selection ： except gate Places like that , You need to limit the output to 0-1 outside , Try not to use sigmoid, It can be used tanh perhaps relu Activation functions like that .1. sigmoid Function in -4 To 4 Section in , There's a big gradient . Outside the range , The gradient is close to 0, It's easy to cause the gradient to disappear .2. Input 0 mean value ,sigmoid The output of the function is not 0 Mean .

6. Loss function round plus regular , A round without regularity

7. The choice of optimizer ：adam,adadelta etc. , On small data , The effect of the experiment is not as good as sgd, sgd The convergence rate will be slower , But the final result of convergence , It's generally better . If you use sgd Words , You can choose from 1.0 perhaps 0.1 The learning rate started to , After a while , Check on the validation set , If cost No decline , Cut the learning rate by half . Many papers do this , The results of the experiment are also very good . Of course , You can also use ada The series starts with , At the end of the day , Replace it with sgd Keep training . There will also be improvements . It is said that adadelta In general, the effect of classification is better ,adam In the generation problem, the effect is better .

8. ensemble

   • The same parameters , Different initialization methods

   • Different parameters , adopt cross-validation, Choose the best groups

     k Detailed explanation of folding and crossing ：https://www.cnblogs.com/henuliulei/p/13686046.html

   • The same parameters , Different stages of model training , That is, models with different iterations .

   • Different models , Linear fusion . for example RNN And traditional models .