Error back propagation

1、 Activate the implementation of the function layer

1.1 Relu Back propagation implementation

The code is as follows ：

class Relu():
    def __init__(self):
        self.x = None

    def forward(self, x):
        self.x = np.maximum(0,x)
        out = self.x

        return out

    # dout  Is the derivative from the upper layer 
    def backward(self, dout):
        dx = dout
        dx[self.x <=0] =0
        return dx

1.2 Sigmoid Back propagation implementation

The code is as follows ：

class _sigmoid:
    def __init__(self):
        self.out = None
    
    def forward(self,x):
        out = 1/(1+np.exp(-x))
        self.out = out
        return out
    
    def backward(self, dout):
        dx = dout * self.out *(1-self.out)
        return dx

2、Affine The realization of the layer ( Fully connected layer )

Affine The general form of layer is , among x It's layer input ,w Is the parameter ,b Is an offset ,f It's a nonlinear activation function .

about X( matrix ) The derivation of , You can refer to the following formula , It should be noted that ,X and   The same shape ,W and   The same shape

                        ​​​​​​​        ​​​​​​​        ​​​​​​​        ​​​​​​​    ​​​​​ 

The code implementation is as follows ：

class Affine:
    def __init__(self, W, b):
        self.W = W
        self.b = b
        self.x = None
        self.dW = None
        self.db = None
    def forward(self,x):

        self.x = x
        out = np.dot(x,self.W) + self.b
        return out
    def backward(self, dout):
        dx = np.dot(dout, self.W.T)
        self.dW = np.dot(self.x.T, dout)
        self.db = np.sum(dout, axis=0)
        return dx

3、Softmaxwithloss The realization of the layer

Suppose the output of the last layer of the network is z, after Softmax Post output is p, The real label is y(ont-hot code ）, among C Express C Categories , So the loss function is ：

        ​​​​​​​        ​​​​​​​        ​​​​​​​        ​​​​​​​        ​​​​​​​        ​​​​​​​        ​​​​​​​    

because p yes z after Softmax Output after function calculation , namely p=softmax(z), among

Loss Yes z The derivative of is ：

The code implementation is as follows ：

class SoftmaxWithLoss():
    def __init__(self):
        self.loss = None
        self.p = None
        self.y = None
    
    def forward(self,x,y):
        self.y = y
        self.p = softmax(x)
        self.loss = cross_entropy_error(self.p,self.y)
        return self.loss
    
    def backward(self, dout= 1):
        batch_size = self.y.shape[0]
        dx = (self.p - self.y) / batch_size
        return dx