SimCLR

SimCLRRepresentations are learned by maximizing the consistency of the same data under different augmentations through a contrastive loss in the hidden space.SimCLRThe framework has four main components,分别是：数据增广,encode网络,projection headNetworks and Contrastive Learning Functions.

对于数据$x$,Extract two independent data augmentation operators from the same data augmentation family（$t\sim T$和$t^{\prime }\sim T$）,to get two related views${\hat{x}}_i$和${\hat{x}}_j$,${\hat{x}}_i$和${\hat{x}}_j$是一对正样本,Then a neural network encoder$f\left(\cdot \right)$Extract features from augmented data$h_i=f\left({\hat{x}}_i\right),h_j=f\left({\hat{x}}_j\right),$.And then a small neural networkproject head $g\left(\cdot \right)$Map features into the space of contrastive losses.project headwith one hidden layerMLP获取$z_i=g\left(h_i\right)=W^{\left(2\right)}\sigma \left(W^{\left(1\right)}{h}_i\right)$.

For contains a pair of positive samples${\hat{x}}_i$和${\hat{x}}_j$的集合$\{{\hat{x}}_k\}$,The contrast prediction task aims for a given${\hat{x}}_i$在$\{\hat{x}{\}}_{k\ne i}$中识别出${\hat{x}}_j$.随机挑选$N$个样本组成一个minibatch,这个minibatch中则有$2N$个数据样本,将其他$2\left(N-1\right)$an amplified sample as thisminibatch中的负样本,设$sim\left(u,v\right)=u^{T}v/\Vert u\Vert \Vert v\Vert$表示$l_2$Yours after regularization$u$和$v$的点积,Then for a pair of positive samples$\left(i,j\right)$,The loss function is defined as follows：

$$l_{i,j}=-log\frac{exp\left(sim\left(z_i,z_j\right)/\tau \right)}{{\sum }_{k=1}^{2N}{1}_{[k\ne i]}exp\left(sim\left(z_i,z_k\right)/\tau \right)}$$

The final loss function computes aminibatchAll positive sample pairs in ,包括$\left(i,j\right)$和$\left(j,i\right)$.下面是SimCLR的伪代码.从伪代码中可以看出,编码器$f\left(\cdot \right)$和project head $g\left(\cdot \right)$ Parameters are updated during training,But only the encoder$f\left(\cdot \right)$用于下游任务.

simCLR不采用memory bank的形式进行训练,rather increasebatchsize,bacth size为8192,对于每一个正样本,将会有16382Instances of negative samples.增大batch sizeActually equivalent to eachminibatchdynamically generate onememory bank.The papers found using standard onesSGD/Momentum,大batch sizeIt is unstable during training,论文中采用LARS优化器.

参考

1. The Illustrated SimCLR Framework
2. SimCLR