大型语言模型在广泛的任务中显示出令人印象深刻的小样本结果。但是，当知识是此类结果的关键时，例如问答和事实检查等任务的情况下，似乎需要大量参数计数来存储知识。众所周知，检索增强模型在不需要太多参数的情况下擅长知识密集型任务，但尚不清楚它们是否适用于小样本设置。在这项工作中，我们展示了 Atlas，这是一种精心设计和预训练的检索增强语言模型，能够通过很少的训练示例学习知识密集型任务。我们对广泛的任务进行评估，包括 MMLU、KILT 和 NaturalQuestions，并研究文档索引内容的影响，表明它可以轻松更新。值得注意的是，Atlas 仅使用 64 个示例就达到了 42% 以上的自然问题准确率，尽管参数减少了 50 倍，但其性能比 540B 参数模型高出 3%。
Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3% despite having 50x fewer parameters.
