Language Models are Unsupervised Multitask Learners

← Back to document page

Language Models are Unsupervised Multitask LearnersAlec Radford*1Jeffrey Wu*1Rewon Child1David Luan1Dario Amodei**1Ilya Sutskever**1AbstractNatural Language processing tasks, such as ques-tion answering, machine translation, reading com-prehension, and summarization, are typicallyapproached with supervised learning on task-specific datasets. We demonstrate that languagemodels begin to learn these tasks without any ex-plicit supervision when trained on a new datasetof millions of webpages called WebText. Whenconditioned on a document plus questions, the an-swers generated by the Language model reach 55F1 on the CoQA dataset - matching or exceedingthe performance of 3 out of 4 baseline systemswithout using the 127,000+ training capacity of the Language model is

the promise of language models to perform specific tasks, such as commonsense reasoning (Schwartz et al.,2017) and sentiment analysis (Radford et al.,2017). In this paper, we connect these two lines of work and con-tinue the trend of more general methods of transfer. We demonstrate language models can perform down-stream

  Language, Paper, Multitask