Language Models are Unsupervised Multitask Learners

Language Models are Unsupervised Multitask LearnersAlec Radford*1 Jeffrey Wu*1 Rewon Child1 David Luan1 Dario Amodei**1 Ilya Sutskever**1 AbstractNatural Language processing tasks, such as ques-tion answering, machine translation, reading com-prehension, and summarization, are typicallyapproached with supervised learning on task-specific datasets. We demonstrate that languagemodels begin to learn these tasks without any ex-plicit supervision when trained on a new datasetof millions of webpages called WebText. Whenconditioned on a document plus questions, the an-swers generated by the Language model reach 55F1 on the CoQA dataset - matching or exceedingthe performance of 3 out of 4 baseline systemswithout using the 127,000+ training capacity of the Language model is essentialto the success of zero-shot task transfer and in-creasing it improves performance in a log-linearfashion across tasks.

Language Models are Unsupervised Multitask Learners to infer and perform many different tasks on examples with this type of format. Language modeling is also able to, in principle, learn the tasks ofMcCann et al.(2018) without the need for explicit supervision of …

Loading..

Tags:

  Language, Model, Learners, Unsupervised, Language models are unsupervised multitask learners, Multitask

Information

Related search queries