Language Models are Unsupervised Multitask Learners

Language Models are Unsupervised Multitask LearnersAlec Radford*1 Jeffrey Wu*1 Rewon Child1 David Luan1 Dario Amodei**1 Ilya Sutskever**1 AbstractNatural Language processing tasks, such as ques-tion answering, machine translation, reading com-prehension, and summarization, are typicallyapproached with supervised learning on task-specific datasets. We demonstrate that languagemodels begin to learn these tasks without any ex-plicit supervision when trained on a new datasetof millions of webpages called WebText. Whenconditioned on a document plus questions, the an-swers generated by the Language model reach 55F1 on the CoQA dataset - matching or exceedingthe performance of 3 out of 4 baseline systemswithout using the 127,000+ training capacity of the Language model is essentialto the success of zero-shot task transfer and in-creasing it improves performance in a log-linearfashion across tasks. Our largest model , GPT-2,is a parameter Transformer that achievesstate of the art results on 7 out of 8 tested lan-guage modeling datasets in a zero-shot settingbut still underfits WebText.

Language Models are Unsupervised Multitask Learners Alec Radford * 1Jeffrey Wu Rewon Child David Luan 1Dario Amodei ** Ilya Sutskever ** 1 Abstract Natural language processing tasks, such as ques-tion answering, machine translation, reading com-

Loading..

Tags:

  Language, Model, Learners, Unsupervised, Language models are unsupervised multitask learners, Multitask

Information

Related search queries