arXiv:2106.07139v2 [cs.AI] 15 Jun 2021

1 Pre-Trained Models: Past, present and future Xu Han1 , Zhengyan Zhang1 , Ning Ding1 , Yuxian Gu1 , Xiao Liu1 , Yuqi Huo2 , Jiezhong Qiu1 , Yuan Yao1 , Ao Zhang1 , Liang Zhang2 , Wentao Han1 , Minlie Huang1 , Qin Jin2 , Yanyan Lan4 , Yang Liu1,4 , Zhiyuan Liu1 , Zhiwu Lu3 , Xipeng Qiu5 , Ruihua Song3 , Jie Tang1 , Ji-Rong Wen3 , Jinhui Yuan6 , Wayne Xin Zhao3 , Jun Zhu1 . 1. Department of Computer Science and Technology, Tsinghua University, Beijing, China 2. School of Information, Renmin University of China, Beijing, China 3. Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China 4. Institute for AI Industry Research, Tsinghua University, Beijing, China 5. School of Computer Science, Fudan University, Shanghai, China 6. OneFlow Inc., Beijing, China [ ] 11 Aug 2021. Abstract 1 Introduction Large-scale pre-trained models (PTMs) such Deep neural networks, such as convolutional neural as BERT and GPT have recently achieved networks (CNNs) (Krizhevsky et al.)

2 , 2012; Kim, great success and become a milestone in the 2014; Kalchbrenner et al., 2014; He et al., 2016), field of artificial intelligence (AI). Owing to recurrent neural networks (RNNs) (Sutskever sophisticated pre-training objectives and huge model parameters, large-scale PTMs can ef- et al., 2014; Donahue et al., 2015; Liu et al., fectively capture knowledge from massive la- 2016; Wu et al., 2016), graph neural networks beled and unlabeled data. By storing knowl- (GNNs) (Kipf and Welling, 2016; Velic kovic et al., edge into huge parameters and fine-tuning on 2018; Schlichtkrull et al., 2018), and attention neu- specific tasks, the rich knowledge implicitly ral networks (Jaderberg et al., 2015; Wang et al., encoded in huge parameters can benefit a vari- 2017), have been widely applied for various artifi- ety of downstream tasks, which has been exten- cial intelligence (AI) tasks in recent years.

3 Differ- sively demonstrated via experimental verifica- ent from previous non-neural models that largely tion and empirical analysis. It is now the con- sensus of the AI community to adopt PTMs relied on hand-crafted features and statistical meth- as backbone for downstream tasks rather than ods, neural models can automatically learn low- learning models from scratch. In this paper, dimensional continuous vectors ( , distributed we take a deep look into the history of pre- representations) from data as task-specific features, training, especially its special relation with thereby getting rid of complex feature engineer- transfer learning and self-supervised learning, ing. Despite the success of deep neural networks, a to reveal the crucial position of PTMs in the AI. number of studies have found that one of their crit- development spectrum.

4 Further, we compre- hensively review the latest breakthroughs of ical challenges is data hungry. Since deep neural PTMs. These breakthroughs are driven by the networks usually have a large number of param- surge of computational power and the increas- eters, they are thus easy to overfit and have poor ing availability of data, towards four impor- generalization ability (Belkin et al., 2019; Xu et al., tant directions: designing effective architec- 2021) without sufficient training data. tures, utilizing rich contexts, improving com- Considering this issue, over the same period of putational efficiency, and conducting interpre- developing deep neural networks, massive efforts tation and theoretical analysis. Finally, we dis- cuss a series of open problems and research have been devoted to manually constructing high- directions of PTMs, and hope our view can in- quality datasets for AI tasks (Deng et al.)

5 , 2009; Lin spire and advance the future study of PTMs. et al., 2014; Bojar et al., 2014), making it possible to learn effective neural models for specific tasks The first six authors contribute equally to organize this paper. The order is determined by dice rolling. that are superior to conventional non-neural models.. All faculty authors are alphabetically sorted. However, it is expensive and time-consuming to 90 Human ( ) 90 Human ( ). 80 80 Interactive SSA (%). 70. Score of GLUE (%). 70. 60 60. 50 50 40 40. 30 30. 20 20. GPT BERT Roberta ELECTRA T5 DEBERTA XiaoIce DialoGPT Cleverbot Mitsuku Meena (base) Meena (2018) (2018) (2019) (2020) (2020) (2021) (2018) (2018) (2019) (2020) (2020) (2021). (a) Evaluation on language understanding benchmark GLUE. (b) Manual evaluation on dialogue systems. Figure 1: The two figures show the significant improvement on performance of both language understanding and language generation after using large-scale PTMs.

6 Manually annotate large-scale data. For example, 2012; Simonyan and Zisserman, 2015; Szegedy utilizing crowdsourcing to segment images costs et al., 2015; He et al., 2016) are pre-trained on the about $ per image (Liu et al., 2020b). Some human-annotated visual recognition dataset Ima- complex tasks that require expert annotations may geNet (Deng et al., 2009). Benefiting from the charge much more to build their datasets. Several strong visual knowledge distributed in ImageNet, tasks such as visual recognition (Deng et al., 2009) fine-tuning these pre-trained CNNs with a small and machine translation (Bojar et al., 2014) have amount of task-specific data can perform well on datasets containing millions of samples, yet it is downstream tasks. This triggers the first wave of impossible to build such large-scale datasets for all exploring pre-trained models (PTMs) in the era of AI tasks.

7 More generally, the dataset of a specific deep learning. In this wave, PTMs are used for al- AI task usually has a limited size. Hence, for a most all CV tasks such as image classification (He long time until now, it has been a key research et al., 2016), object detection (Sermanet et al., issue: how to train effective deep neural models for 2014; Ren et al., 2016), image segmentation (Long specific tasks with limited human-annotated data. et al., 2015), and image captioning (Vinyals et al., 2015). One milestone for this issue is the introduction of transfer learning (Thrun and Pratt, 1998; Pan The natural language processing (NLP) com- and Yang, 2009). Instead of training a model from munity was also aware of the potential of PTMs scratch with large amounts of data, human beings and started to develop PTMs for NLP tasks (Qiu can learn to solve new problems with very few sam- et al.)

8 , 2020). To take full advantage of large- ples. This amazing learning process is motivated scale unlabeled corpora to provide versatile lin- by the fact that human beings can use previously guistic knowledge for NLP tasks, the NLP com- learned knowledge to handle new problems. In- munity adopts self-supervised learning (Liu et al., spired by this, transfer learning formalizes a two- 2020b) to develop PTMs. The motivation of self- phase learning framework: a pre-training phase to supervised learning is to leverage intrinsic correla- capture knowledge from one or more source tasks, tions in the text as supervision signals instead of and a fine-tuning stage to transfer the captured human supervision. For example, given the sen- knowledge to target tasks. Owing to the wealth tence Beijing is the capital of China , we mask of knowledge obtained in the pre-training phase, the last word in the sentence, and then require mod- the fine-tuning phase can enable models to well els to predict the masked position with the word handle target tasks with limited samples.

9 China . Through self-supervised learning, tremen- Transfer learning provides a feasible method for dous amounts of unlabeled textual data can be uti- alleviating the challenge of data hungry, and it has lized to capture versatile linguistic knowledge with- soon been widely applied to the field of computer out labor-intensive workload. This self-supervised vision (CV). A series of CNNs (Krizhevsky et al., setting in essence follows the well-known language 2. 4000 100000 103. Publications Model Size 106 Data Size 3500 Citations 80000. 3000. 105. 2500. Model Size (M). Data Size (GB). 60000 102. Publications Citations 2000 104. 40000. 1500. 1000 103. 20000 101. 500. 102. 0 0. 1990 1995 2000 2005 2010 2015 2020 GPT (2018) BERT (2018) GPT-2 (2019) RoBERTa (2019) T5 (2019) GPT-3 (2020) Switch (2021). Year Model (Year). (a) The number of publications on language models and their (b) The model size and data size applied by recent NLP PTMs.)

10 Citations in recent years. A base-10 log scale is used for the figure. Figure 2: Figure 2(a) shows the number of publications with the keyword language model as well as their citations in different years. Figure 2(b) shows the parameter size of large-scale PTMs for NLP tasks and the pre- training data size are increasing by 10 times per year. From these figures, we can find that, after 2018, when large-scale NLP PTMs begin to be explored, more and more efforts are devoted to this field, and the model size and data size used by the PTMs are also getting larger. model learning (Bengio et al., 2003). structures, as well as factual knowledge from the For a long time, the problem of vanishing or ex- text. By fine-tuning large-scale PTMs with quite ploding gradients (Bengio et al., 1994) is the pain a few samples, rich linguistic knowledge of PTMs point of using deep neural networks for NLP tasks.