This blog is mainly about self supervised learning （Self-Supervised Learning）

introduce

Here are some examples of self supervised learning models

BERT Model 340M parameters
ELMO Model 94M parameters
GPT-2 Model 1542M parameters
GPT-3 Model 175M parameters
Megatron Model 8B parameters
Transformer Model 1.6T parameters
…

Self monitoring definition

BERT

BERT Enter a line of vectors , Output another line of vector


BERT To learn mask Places and “ bay ” It's the same category
BERT+Linear Training together

however Next Sentence Prediction It doesn't seem to work ？

BERT You can also do many downstream tasks , Later on

GLUE

In order to make NLU（ natural language understanding ） The task plays the greatest role , From New York University 、 Institutions such as the University of Washington have created a multi task natural language understanding benchmark and analysis platform , That is to say GLUE（General Language Understanding Evaluation）GLUE Nine tasks involve natural language inference 、 The text contains 、 Sentiment analysis 、 Semantic similarity and other tasks . image BERT、XLNet、RoBERTa、ERINE、T5 And other well-known models will be tested on this benchmark .

Case

pre-train That is, to train students to fill in the blanks BERT

BERT yes semi-supervised Of . Because in downstream tasks , Need marked information ; But do self-supervised When , It's not marked .


case2 Follow case1 The difference is , stay pre-train When , A set of parameters has been initialized .

Other related

Why does BERT work

GPT series

other

self-supervised It can be used not only in text , It can also be used in image and voice .





self-supervised