The value of BatchSize is not the bigger the better.

Usually we may think that the training effect of the model will be better when setting a larger batchsize.The reasons are as follows:

1. Since the model obtains more training data each time, the descending direction of the model will be more accurate, and the model training curve will be smoother.

2. Reduced training time.At the same epoch, the number of batches required by batchsize is reduced, so the processing speed becomes faster.

But ah but,

Larger batchsize has the following issues to be aware of:

1. Memory problem.Large batches may cause memory/video memory overflow

2. The generalization ability decreases.This is something I hadn't considered before.Using a batch size that is too large may negatively affect the accuracy of the network during training, as it reduces the randomness of gradient descent.

Using a smaller batch size produces more erratic, more random weight updates.This has two positive effects.First, it can help the training "jump out" of local minima that it may have gotten stuck in before, and second, it can stabilize the training to a "flatter" minimum, which usually indicates better generalization performance.

HowSelect the Batch size when training the neural network? - Knowing (zhihu.com)

The above link (invasion and deletion) states:

• When there are enoughHashrate, select a batch size of 32 or less.
• When the computing power is not enough, make a trade-off between efficiency and generalization, and try to choose a smaller batch size.
• When the model is trained to the end, if you want to improve the performance in a more refined way (such as the paper experiment/competition to the end), there is a useful trick, which is to set the batch size to 1, that is, do pure SGD, and slowly reduce the error.