Pytoch learning record (x): data preprocessing + batch normalization (BN)

Data preprocessing

At present, the most common method of data preprocessing is centralization and standardization , Centralization is equivalent to correcting the central position of data , The implementation is very simple , It is to subtract the corresponding mean value from each feature dimension , Finally get 0 mean value （） The pixel range is [-128,127], With 0 Centered Characteristics of . Standardization is also very simple , When the data becomes 0 After mean , In order to make different feature dimensions have the same scale , It can be divided by the standard deviation to approximate a normal distribution , It can also be converted to... Based on the maximum and minimum values -1 ~ 1 Between .

Batch Normalization The batch （BN）

Batch of standardized , In short , It is the output of each layer of network , Make a normalization of it , Make it obey the standard normal distribution , In this way, the input of the latter layer network is also a standard normal distribution , So I can train better , Speed up convergence .

The first and second lines calculate a batch Mean and variance of data in , Then use the third formula to batch Standardize each data point in the , 𝜖 A small constant introduced to calculate stability , Usually take 10−5 , Finally, the final output result is obtained by weight correction , Very simple , Now we can realize a simple one-dimensional case , That is, the situation in neural networks
https://www.cnblogs.com/yongjieShi/p/9332655.html

https://zhuanlan.zhihu.com/p/69431151

nn.BatchNorm2d

from torch import nn
import torch

m = nn.BatchNorm2d(3)  # bn The parameters set are actually channel Parameters of 
input = torch.randn(4, 3, 2, 2)
output = m(input)