view() and reshape()

Written in the beginning ：
There is a big man's summary that is very in place ： Blog

summary

1. view() In operation tensor when , need tensor Memory is continuous , And when changing the size ,view() The operation will not open up new memory space . But make sure that tensor continuity , Yes tensor Conduct tensor.contiguous() when , Will open up New memory space , Store continuous data in memory .
2. reshape() operation , And view() The effect is as like as two peas. , But it's better than view() The more advanced , Operated tensor When memory is continuous , Direct adoption reshape Will not open up new memory ; Operated tensor When memory is not continuous ,reshape Operation will open up new memory , Right again tensor Conduct reshape.
3. Last , use reshape operation Just a matter of

expand() and repeat()

expand()

Returns the tensor after the current tensor is expanded in a certain dimension . Expand （expand） The tensor does not allocate new memory , Just create a new view on the existing tensor （view）, A size equal to 1 Expand the dimension of to a larger size
Example ：

import torch
x = torch.tensor([1, 2, 3])
x.expand(2,3)
tensor([[1, 2, 3],
			[1, 2, 3]])

Be careful expand（） The dimension can only be extended to 1 Dimension of , Dimension is not 1 Keep the parts consistent

repeat()

Repeat the tensor along a specific dimension , and expand() The difference is , This function copies the tensor data
Example

import torch

x = torch.tensor([1, 2, 3])
x.repeat(3,2)
tensor([[1, 2, 3, 1, 2, 3],
		[1, 2, 3, 1, 2, 3],
		[1, 2, 3, 1, 2, 3]])

x2 = torch.randn(2, 3, 4)
x2.repeat(2, 1, 3).shape

torch.Tensor([4, 3, 12])

Multiplication operation

pytorch The multiplication operations in are ：torch.mm(), torch.bmm(), torch.matmul(), torch.mul(), Operator , as well as torch.einsum()

Two dimensional matrix multiplication torch.mm()

This function is generally only used to calculate the matrix multiplication of two two-dimensional matrices , And does not support broadcast operation .
torch.mm(mat1, mat2, out=None), among mat1 by (nxm),mat2 by (mxd), The output dimension is (nxd)

Three dimensional band batch Matrix multiplication of torch.bmm()

The two inputs of the function must be a three-dimensional matrix and the first dimension is the same （ Express Batch dimension ）, I won't support it broadcast operation .

Because neural network training generally adopts mini-batch, Often input three-dimensional band batch Matrix , So provide torch.bmm(bmat1, bmat2, out=None), among bmat1 by (bxnxm),bmat2 by (bxmxd), Output out The dimension of is (bxnxd)

Multidimensional matrix multiplication torch.matmul()

torch,matmul(input, other, out=None) Support broadcast operation
For multidimensional data matmul() Multiplication , It can be considered that matmul() Multiplication uses the last two dimensions of the two parameters to calculate , Other dimensions can be considered as batch dimension . Suppose the dimensions of the two inputs are input->(100x500x99x11),other->(500x11x99) Then we can think that the multiplication first carries out the matrix multiplication of the last two bits to get (99x11)x(11x99)->(99,99), Then analyze the of the two parameters batch size Respectively (1000x500) and 500, Can be broadcast as (1000x500), So the dimension of the final output is (1000x500x99x99)

Matrix element by element （Element-wise） Multiplication torch.mul()

function torch.mul(mat1, other, out=None), among other Multipliers can be scalars , It can also be a matrix of any dimension , As long as the final multiplication is satisfied broadcast that will do .

Two operators @ and *

1. @： Matrix multiplication , Automatically execute the appropriate matrix multiplication function
2. *：elemnet-wise Multiplication

register_parameter() and parameter()

1. Parameter()

Parameter yes Tensor, namely Tensor All the attributes it has , For example, according to data To access parameter values , use grad To access the parameter gradient

#  Just define a network 
net = nn.Sequential(nn.Linear(4,3), nn.ReLU(), nn.Linear(3,1))
# list Make it accessible 
weight_0 = list(net[0].parameters())[0]
print(weight_0.data)
print(weight_0.grad)

2. register_parameter(name, parameters)
   To build a network module add to parameter
   The biggest difference ：parameter You can register online through name obtain
   Example

class Example(nn.Module):
	def __init__(self):
		super(Example, self).__init__()
		self.W1_params = nn.Parameter(torch.rand(2,3))
		self.register_parameter('W2_params', nn.Parameter(torch.rand(2,3)))
	def forward(self, x):
		return x