A Survey of Visual Transformers

1 1. A Survey of Visual Transformers Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang? , Zhongchao Shi? , Jianping Fan, Zhiqiang He? Abstract Transformer, an attention-based encoder-decoder the Bidirectional Encoder Representations from Transformers model, has already revolutionized the field of natural language (BERT) [5] and its variants [6], [7] serve as auto-encoder processing (NLP). Inspired by such significant achievements, language models built on the Transformer encoders. some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) In the CV field, prior to the Visual Transformers , Con- field, which have demonstrated their effectiveness on three fun- volution Neural Networks (CNNs) have emerged as a dom- [ ] 2 May 2022.

2 Damental CV tasks (classification, detection, and segmentation) inant paradigm [10] [12]. Inspired by the great success of as well as multiple sensory data stream (images, point clouds, such self-attention mechanisms for the NLP tasks [1], [13], and vision-language data). Because of their competitive modeling some CNN-based models attempted to capture the long-range capabilities, the Visual Transformers have achieved impressive performance improvements over multiple benchmarks as com- dependencies through adding a self-attention layer at either pared with modern Convolution Neural Networks (CNNs). In this spatial level [14] [16] or channel level [17] [19], while others Survey , we have reviewed over one hundred of different Visual try to replace the traditional convolutions entirely with the Transformers comprehensively according to three fundamental global [20] or local self-attention blocks [21] [27].

3 Although CV tasks and different data stream types, where a taxonomy Ramachandr et al. have demonstrated the efficiency of self- is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because attention block [24] without the help from CNNs, such pure of their differences on training settings and dedicated vision attention model is still inferior to the State-Of-The-Art (SOTA). tasks, we have also evaluated and compared all these existing CNN models on the prevailing benchmarks. Visual Transformers under different configurations. Furthermore, With the grateful achievements of linguistic Transformers we have revealed a series of essential but unexploited aspects and the rapid development of Visual attention-based models, that may empower such Visual Transformers to stand out from numerous architectures, , slack high-level semantic embed- numerous recent works have migrated the Transformers to the dings to bridge the gap between the Visual Transformers and CV tasks, and some comparable results have been achieved.

4 The sequential ones. Finally, three promising research direc- Cordonnier et al. [28] theoretically demonstrated the equiva- tions are suggested for future investment. We will continue to lence between multi-head self-attention and CNNs, and they update the latest articles and their released source codes at designed a pure Transformer by using patch downsampling and quadratic position encoding to verify their theoretical con- Index Terms Visual Transformer, attention, high-level vision, clusion. Dosovitskiy et al. [29] further extended such a pure 3D point clouds, multi-sensory data stream, multi-modal, Visual - Transformer for large-scale pre-training, which has achieved linguistic pre-training, self-supervision, neural networks, com- puter vision.

5 SOTA performance over many benchmarks. Additionally, the Visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], I. I NTRODUCTION tracking [32], generation [33], and enhancement [34]. T RANSFORMER [1], which adopts an attention-based structure, has first demonstrated its tremendous effects on the tasks of sequence modeling and machine transla- As illustrated in Fig. 1, following the pioneer works [29], [30], hundreds of Transformer-based models have been pro- posed for various vision applications within the last year. Thus, tion. As illustrated in Fig. 1, Transformers have gradually a systematic literature Survey is strongly desired to identify, emerged as the predominant deep learning models for many categorize, and evaluate the performance of these existing NLP tasks.

6 The most recent dominant models are the self- Visual Transformers . Considering that the readers may come supervised Transformers , which are pre-trained over sufficient from different areas, we review all these Visual Transformers datasets and then fine-tuned over a small sample set for a according to three fundamental CV tasks ( , classification, given downstream task [2] [9]. The Generative Pre-trained detection, and segmentation) and data stream types ( , im- Transformer (GPT) families [2] [4] leverage the Transformer age, point clouds, multi-stream data). As illustrated in Fig. 3, decoders to enable auto-regressive language modeling, while this Survey categorizes all these existing methods into multiple groups according to their dedicated vision tasks, data stream This work was done at AI Lab, Lenovo Research.

7 Yang Liu, Yao Zhang, Yixin Wang, and Feng Hou are with Institute types, motivations, and structural characteristics. of Computing Technology, Chinese Academy of Sciences, Beijing, 100000, Before us, several reviews on the Transformers have been China and also with University of Chinese Academy of Sciences, Beijing, published, where Tay et al. [45] reviewed the efficiency of the 100000, China (E-mail: Jin Yuan is with Southeast University, Nanjing, 214135, China. linguistic Transformers , Khan et al. [46] and Han et al. [47]. Jiang Tian, Zhongchao Shi, and Jianping Fan are with AI Lab, Lenovo summarized the early Visual Transformers and attention-based Research, Beijing, 100000, China.)

8 Models. The most recent review of the Transformers is intro- Yang Zhang, Zhiqiang He are with Lenovo Ltd., Beijing, 100000, China. ? Corresponding authors: Zhiqiang He; Zhongchao Shi; Yang Zhang (E- duced by Lin et al., which provides a systematic review of mail: {hezq; shizc2; various Transformers , but they only mention vision applica- 2. Growth of Transformer Citations Vs. Half Year Growth of ViT Citations Vs. Month Number of Citations Number of Citations 200 100. 160 80. 120 60. 80 40. 40 20. Transformer GPT BERT XL-Net Roberta ALBERT GPT-3 ViT DeiT PVT Swin CaiT Refiner VOLO. 2017 Oct. 2018 Jun. 2018 Oct. 2019 Jun. 2019 Jul.)}

9 2020 Feb. 2020 Jun. 2020 Oct. 22 2021 Jan. 15 2021 Feb. 24 2021 Mar. 25 2021 Apr. 7 2021 Jun. 7 2021 Jun. 28. Auto-Encoder Language Model Backbone SOTA on ImageNet-1k CvT ViT-G. with external data 2021 Mar. 29 2021 Jun. 8. Backbone SOTA on ImageNet-1k LV-ViT CoAtNet Auto-Regressive Language Model with no external data GPT-2. 2019 Oct. 2021 Apr. 23 2021 Jun. 9. Fig. 1. Odyssey of Transformer application & Growth of both Transformer [1] and ViT [29] citations according to Google Scholar. (Upper Left) Growth of Transformer citations in multiple conference publication including: NIPS, ACL, ICML, IJCAI, ICLR, and ICASSP. (Upper Right) Growth of ViT citations in Arxiv publications.

10 (Bottom Left) Odyssey of language model [1] [8]. (Bottom Right) Odyssey of Visual Transformer backbone where the black [29], [35] [39] is the SOTA with external data and the blue [40] [44] refers to the SOTA without external data (best viewed in color). tions sketchily [48]. Distinctively, this paper aims to provide Sec. VI clarifies the mainstream and its variants for the Visual more comprehensive review of the most-recently Visual Trans- Transformers in the segmentation field according to their em- formers and categorize them systematically: bedding forms ( , patch embedding and query embedding). (1) Comprehensiveness & Readability.