NLP-Reading Comprehension Task Learning Summary Overview

Reading comprehension task general form

• Cloze: Given an article, mask out part of the article, and then let the model fill in according to the context
  • The form of reading comprehension is very similar to pre-training, so it is very suitable for pre-training models
  • The cloze model can be thought of as maximizing the conditional probability P(a|C-a), where a is the removed content and C is the context
  • Usually, the probability distribution output (attention value) is performed on the candidate dictionary, and the maximum value is obtained as the result after normalization
  • Limited application scenarios
• Multiple choice: Given an article, a question, and multiple alternative answers, the model selects the correct answer from multiple alternative answers
  • Multiple choice model, which can be seen as maximizing probability P(a|C,Q,A), where a is the correct answer, C is the context, Q is the question, and A is the list of candidate answers
  • Similar to the two-tower model, one side is the input article and question, the other side is the candidate answer, and the final output probability value is the highest
  • There are often no candidate answers in actual scenarios, and the limitations are very large
• Segment extraction: Given an article and a question, let the model extract a continuous sequence from the article as the answer to the question
  • Segment extraction task, treat the article as a collection of different sequences, select consecutive subsequences as the answer, and the output is generally the start and end indexes
  • The model can be seen as maximizing the probability P(a|C,Q), where C is the context (article), Q is the question, and a is the set of extracted sequences
• Freedom generation: Given an article and a question, let the model generate a sequence to answer the question, not limited to sentences in the article
  • The smoothness and correctness of freely generated content is difficult

Reading comprehension assessment indicators

• Accuracy: The proportion of correct results in all prediction results
  • There are standard answers for cloze and multiple choice, so accuracy is generally used to judge
• EM(Exact Match): The predicted result matches the correct result, and the exact match is counted as 1, the percentage of the calculation
  • Commonly used for extractive reading comprehension
• F1: Harmonic mean of precision and recall
  • Commonly used for extractive reading comprehension
• BLEU-4: n-gram based evaluation metrics
  • Commonly used in generative reading comprehension
• Rouge-L: Quasi-call weighted harmonic mean based on longest common subsequence
  • It is often used to evaluate the quality of automatic summarization, where L represents the longest common subsequence
  • Commonly used in generative reading comprehension

General model structure for reading comprehension

• Embedding Layer
  • Used to embed the input data
  • Common methods include pre-training, word vector technology
• Feature Layer
  • This layer is mainly to obtain the context feature representation
  • Common lstm, cnn, transformer
• Interaction Layer
  • Realize the interaction between paragraph and paragraph, content and question, question and answer
  • Common way (one-way/two-way) attention mechanism
• Output layer
  • Generates the final result
  • Close only needs to output the value with the highest probability distribution or highest score
  • Multiple-choice tasks require scoring candidate answers to get the highest-scoring answer
  • The extraction method needs to match the sequence combination most relevant to the problem, and output the start and end index
  • Generative tasks need to be generated freely, similar to GPT-2