Timing model: gated cyclic unit network (Gru)

1. Model definition

Gated loop unit network （Gated Recurrent Unit,GRU）¹ Is in LSTM A simplified variant developed on the basis of , It can usually achieve the same speed as LSTM The effect of the model is similar ².

2. Model structure and forward propagation formula

GRU The hidden state calculation module of the model does not introduce additional memory units , The logic gate is simplified to Reset door （reset gate） and Update door （update gate）, Its structural diagram and forward propagation formula are as follows ：

$$\{\begin{array}{llc}transportEnter\; into：& X_t\in R^{m\times d},H_{t-1}\in R^{m\times h}& \\ & & \\ heavySet\; updoor：& R_t=\sigma (X_t{W}_{xr}+H_{t-1}{W}_{hr}+b_r),& W_{xr}\in R^{d\times h},W_{hr}\in R^{h\times h}\\ & & \\ Houchooseimplicithiddenshapestate：& {\tilde{H}}_t=tanh(X_t{W}_{xh}+(R_t\odot H_{t-1})W_{hh}+b_h),& W_{xh}\in R^{d\times h},W_{hh}\in R^{h\times h}\\ & & \\ morenewdoor：& Z_t=\sigma (X_t{W}_{xz}+H_{t-1}{W}_{hz}+b_z),& W_{xz}\in R^{d\times h},W_{hz}\in R^{h\times h}\\ & & \\ implicithiddenshapestate：& H_t=Z_t\odot H_{t-1}+(1-Z_t)\odot {\tilde{H}}_t& \\ & & \\ transportOut：& {\hat{Y}}_t=H_t{W}_{hy}+b_y,& W_{hy}\in R^{h\times q}\\ & & \\ damagelossLetterCount：& L={\sum }_{t=1}^{T}l(\widehat{(}Y{)}_t,Y_t)& \end{array}\text{\hspace{1em}(2.1)}$$

3. GRU Back propagation process

Because no additional memory units are introduced , therefore GRU Back propagation calculation diagram and RNN Agreement （ Such as the author's article ： Time series model ： Cyclic neural network （RNN） Chinese 3 Shown ）,GRU The back propagation formula is as follows ：

$$\frac{\partial L}{\partial {\hat{Y}}_t}=\frac{\partial l({\hat{Y}}_t,Y_t)}{T\cdot \partial {\hat{Y}}_t}\text{\hspace{1em}(3.1)}$$

$$\frac{\partial L}{\partial {\hat{Y}}_t}\Rightarrow \{\begin{array}{l}\frac{\partial L}{\partial W_{hy}}=\frac{\partial L}{\partial {\hat{Y}}_t}\frac{\partial {\hat{Y}}_t}{\partial W_{hy}}\\ \\ \frac{\partial L}{\partial H_t}=\{\begin{array}{ll}\frac{\partial L}{\partial {\hat{Y}}_t}\frac{\partial {\hat{Y}}_t}{\partial H_t},& t=T\\ & \\ \frac{\partial L}{\partial {\hat{Y}}_t}\frac{\partial {\hat{Y}}_t}{\partial H_t}+\frac{\partial L}{\partial H_{t+1}}\frac{\partial H_{t+1}}{\partial H_t},& t<T\end{array}\end{array}\text{\hspace{1em}(3.2)}$$

$$\begin{array}{ccccc}\{\begin{array}{l}\frac{\partial L}{\partial W_{xz}}=\frac{\partial L}{\partial Z_t}\frac{\partial Z_t}{\partial W_{xz}}\\ \\ \frac{\partial L}{\partial W_{hz}}=\frac{\partial L}{\partial Z_t}\frac{\partial Z_t}{\partial W_{hz}}\\ \\ \frac{\partial L}{\partial b_z}=\frac{\partial L}{\partial Z_t}\frac{\partial Z_t}{\partial b_z}\end{array}& & & & \{\begin{array}{l}\frac{\partial L}{\partial W_{xh}}=\frac{\partial L}{\partial {\tilde{H}}_t}\frac{\partial {\tilde{H}}_t}{\partial W_{xh}}\\ \\ \frac{\partial L}{\partial W_{hh}}=\frac{\partial L}{\partial {\tilde{H}}_t}\frac{\partial {\tilde{H}}_t}{\partial W_{hh}}\\ \\ \frac{\partial L}{\partial b_h}=\frac{\partial L}{\partial {\tilde{H}}_t}\frac{\partial {\tilde{H}}_t}{\partial b_h}\end{array}\end{array}\text{\hspace{1em}(3.3)}$$

$$\{\begin{array}{l}\frac{\partial L}{\partial W_{xr}}=\frac{\partial L}{\partial R_t}\frac{\partial R_t}{\partial W_{xr}}\\ \\ \frac{\partial L}{\partial W_{hr}}=\frac{\partial L}{\partial R_t}\frac{\partial R_t}{\partial W_{hr}}\\ \\ \frac{\partial L}{\partial b_r}=\frac{\partial L}{\partial R_t}\frac{\partial R_t}{\partial b_r}\end{array}\text{\hspace{1em}(3.4)}$$

And LSTM Empathy ,GRU The key to the solution of the back propagation formula is also the calculation of different time steps （ Pass on ） gradient The solution of , The method is similar to LSTM Consistent, this article will not repeat . And we can also draw qualitative conclusions ,GRU Principles and methods of alleviating long-term dependence LSTM similar , It is realized by adjusting the multiplier of high-order power term and adding low-order power term . among , Resetting the gate helps capture short-term dependencies in the sequence , Update gates help capture long-term dependencies in sequences .（ Please refer to the author's article for details ： Time series model ： Long and short term memory network （LSTM） The proof process in ）

4. Code implementation of the model

4.1 TensorFlow Framework implementations

4.2 Pytorch Framework implementations

import torch
from torch import nn
from torch.nn import functional as F


#
class Dense(nn.Module):
    """ Args: outputs_dim: Positive integer, dimensionality of the output space. activation: Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix. bias_initializer: Initializer for the bias vector. Input shape: N-D tensor with shape: `(batch_size, ..., input_dim)`. The most common situation would be a 2D input with shape `(batch_size, input_dim)`. Output shape: N-D tensor with shape: `(batch_size, ..., units)`. For instance, for a 2D input with shape `(batch_size, input_dim)`, the output would have shape `(batch_size, units)`. """
    def __init__(self, input_dim, output_dim, **kwargs):
        super().__init__()
        #  Hyperparametric definition 
        self.use_bias = kwargs.get('use_bias', True)
        self.kernel_initializer = kwargs.get('kernel_initializer', nn.init.xavier_uniform_)
        self.bias_initializer = kwargs.get('bias_initializer', nn.init.zeros_)
        #
        self.Linear = nn.Linear(input_dim, output_dim, bias=self.use_bias)
        self.Activation = kwargs.get('activation', nn.ReLU())
        #  Parameter initialization 
        self.kernel_initializer(self.Linear.weight)
        self.bias_initializer(self.Linear.bias)

    def forward(self, inputs):
        outputs = self.Activation(
            self.Linear(inputs)
        )
        return outputs


#
class GRU_Cell(nn.Module):
    def __init__(self, token_dim, hidden_dim
                 , reset_act=nn.ReLU()
                 , update_act=nn.ReLU()
                 , hathid_act=nn.Tanh()
                 , **kwargs):
        super().__init__()
        #
        self.hidden_dim = hidden_dim
        #
        self.ResetG = Dense(
            token_dim + self.hidden_dim, self.hidden_dim
            , activation=reset_act, **kwargs
        )
        self.UpdateG = Dense(
            token_dim + self.hidden_dim, self.hidden_dim
            , activation=update_act, **kwargs
        )
        self.HatHidden = Dense(
            token_dim + self.hidden_dim, self.hidden_dim
            , activation=hathid_act, **kwargs
        )

    def forward(self, inputs, last_state):
        last_hidden = last_state[-1]
        #
        Rg = self.ResetG(
            torch.concat([inputs, last_hidden], dim=1)
        )
        Zg = self.UpdateG(
            torch.concat([inputs, last_hidden], dim=1)
        )
        hat_hidden = self.HatHidden(
            torch.concat([inputs, Rg * last_hidden], dim=1)
        )
        hidden = Zg*last_hidden + (1-Zg)*hat_hidden
        #
        return [hidden]

    def zero_initialization(self, batch_size):
        return torch.zeros([batch_size, self.hidden_dim])


#
class RNN_Layer(nn.Module):
    def __init__(self, rnn_cell, bidirectional=False):
        super().__init__()
        self.RNNCell = rnn_cell
        self.bidirectional = bidirectional

    def forward(self, inputs, mask=None, initial_state=None):
        """ inputs: it's shape is [batch_size, time_steps, token_dim] mask: it's shape is [batch_size, time_steps] :return hidden_state_seqence: its' shape is [batch_size, time_steps, hidden_dim] last_state: it is the hidden state of input sequences at last time step, but, attentively, the last token wouble be a padding token, so this last state is not the real last state of input sequences; if you want to get the real last state of input sequences, please use utils.get_rnn_last_state(hidden_state_seqence). """
        batch_size, time_steps, token_dim = inputs.shape
        #
        if initial_state is None:
            initial_state = self.RNNCell.zero_initialization(batch_size)
        if mask is None:
            if batch_size == 1:
                mask = torch.ones([1, time_steps])
            else:
                raise ValueError(' Please give a mask matrix (mask)')

        #  Forward time step cycle 
        hidden_list = []
        hidden_state = initial_state
        for i in range(time_steps):
            hidden_state = self.RNNCell(inputs[:, i], hidden_state)
            hidden_list.append(hidden_state[-1])
            hidden_state = [hidden_state[j] * mask[:, i:i+1] + initial_state[j] * (1-mask[:, i:i+1])
                            for j in range(len(hidden_state))]  #  Reinitialize （ Addend function ）
        #
        seqences = torch.reshape(
            torch.unsqueeze(
                torch.concat(hidden_list, dim=1), dim=1
            )
            , [batch_size, time_steps, -1]
        )
        last_state = hidden_list[-1]

        #  Reverse time step cycle 
        if self.bidirectional is True:
            hidden_list = []
            hidden_state = initial_state
            for i in range(time_steps, 0, -1):
                hidden_state = self.RNNCell(inputs[:, i-1], hidden_state)
                hidden_list.append(hidden_state[-1])
                hidden_state = [hidden_state[j] * mask[:, i-1:i] + initial_state[j] * (1 - mask[:, i-1:i])
                                for j in range(len(hidden_state))]  #  Reinitialize （ Addend function ）
            #
            seqences = torch.concat([
                seqences,
                torch.reshape(
                    torch.unsqueeze(
                        torch.concat(hidden_list, dim=1), dim=1)
                    , [batch_size, time_steps, -1])
                ]
                , dim=-1
            )
            last_state = torch.concat([
                last_state
                , hidden_list[-1]
                ]
                , dim=-1
            )

        return {
    
            'hidden_state_seqences': seqences
            , 'last_state': last_state
        }