1 Transformers in Vision: A Survey

1 1 Transformers in Vision: A SurveySalman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir,Fahad Shahbaz Khan, and Mubarak ShahAbstract Astounding results from Transformer models on natural language tasks have intrigued the vision community to study theirapplication to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between inputsequence elements and support parallel processing of sequence as compared to recurrent , Long short-term memory(LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suitedas set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities ( , images, videos,text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and hugedatasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks.

2 This Survey aims toprovide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction tofundamental concepts behind the success of Transformers , self-attention, large-scale pre-training, and bidirectional featureencoding. We then cover extensive applications of Transformers in vision including popular recognition tasks ( , image classification,object detection, action recognition, and segmentation), generative modeling, multi-modal tasks ( , visual-question answering, visualreasoning, and visual grounding), video processing ( , activity recognition, video forecasting), low-level vision ( , imagesuper-resolution, image enhancement, and colorization) and 3D analysis ( , point cloud classification and segmentation). Wecompare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimentalvalue. Finally, we provide an analysis on open research directions and possible future works.

3 We hope this effort will ignite furtherinterest in the community to solve current challenges towards the application of transformer models in computer Terms Self-attention, Transformers , bidirectional encoders, deep neural networks, convolutional networks, INTRODUCTIONTRANSFORMER models [1] have recently demonstratedexemplary performance on a broad range of , text classification, machine translation [2] andquestion answering. Among these models, the most popularones include BERT (Bidirectional Encoder Representationsfrom Transformers ) [3], GPT (Generative Pre-trained Trans-former) v1-3 [4] [6], RoBERTa (Robustly Optimized BERTPre-training) [7] and T5 (Text-to-Text Transfer Transformer)[8]. The profound impact of Transformer models has becomemore clear with their scalability to very large capacity mod-els [9], [10]. For example, the BERT-large [3] model with340 million parameters was significantly outperformed bythe GPT-3 [6] model with 175 billion parameters while thelatest mixture-of-experts Switch transformer [10] scales upto a whopping trillion parameters!

4 The breakthroughs from Transformer networks in Nat-ural Language Processing (NLP) domain has sparked greatinterest in the computer vision community to adapt thesemodels for vision and multi-modal learning tasks (Fig. 1). S. Khan, M. Naseer and F. S. Khan are with the MBZ University ofArtificial Intelligence, Abu Dhabi, : M. Hayat is with the Faculty of IT, Monash University, Clayton VIC3800, Australia. S. W. Zamir is with the Inception Institute of Artificial Intelligence, AbuDhabi, UAE. S. Khan and M. Naseer are also with the CECS, Australian NationalUniversity, Canberra ACT 0200, Australia. F. S. Khan is also with the Computer Vision Laboratory, Link opingUniversity, Sweden. M. Shah is with the Center for Research in Computer Vision, Universityof Central Florida, Orlando, FL 32816, United received March, , visual data follows a typical structure ( , spatialand temporal coherence), thus demanding novel networkdesigns and training schemes.

5 As a result, Transformer mod-els and their variants have been successfully used for imagerecognition [11], [12], object detection [13], [14], segmenta-tion [15], image super-resolution [16], video understanding[17], [18], image generation [19], text-image synthesis [20]and visual question answering [21], [22], among severalother use cases [23] [26]. This Survey aims to cover suchrecent and exciting efforts in the computer vision domain,providing a comprehensive reference to interested architectures are based on a self-attentionmechanism that learns the relationships between elementsof a sequence. As opposed to recurrent networks that pro-cess sequence elements recursively and can only attend toshort-term context, Transformers can attend to completesequences thereby learning long-range relationships. Al-though attention models have been extensively used inboth feed-forward and recurrent networks [27], [28], Trans-formers are based solely on the attention mechanism andhave a unique implementation ( , multi-head attention)optimized for parallelization.

6 An important feature of thesemodels is their scalability to high-complexity models andlarge-scale datasets , in comparison to some of the otheralternatives such as hard attention [29] which is stochastic innature and requires Monte Carlo sampling for sampling at-tention locations. Since Transformers assume minimal priorknowledge about the structure of the problem as comparedto their convolutional and recurrent counterparts [30] [32],they are typically pre-trained using pretext tasks on large-scale (unlabelled) datasets [1], [3]. Such a pre-training avoidscostly manual annotations, thereby encoding highly expres- [ ] 19 Jan 20222 Fig. 1:Statistics on the number of times keywords such as BERT, Self-Attention, and Transformers appear in the titles of Peer-reviewed and arXiv papers over the past few years (in Computer Vision and Machine Learning). The plots show consistent growthin recent literature. This Survey covers recent progress on Transformers in the computer vision and generalizable representations that model rich rela-tionships between the entities present in a given dataset.

7 Thelearned representations are then fine-tuned on the down-stream tasks in a supervised manner to obtain paper provides a holistic overview of the trans-former models developed for computer vision develop a taxonomy of the network design space andhighlight the major strengths and shortcomings of the ex-isting methods. Other literature reviews mainly focus onthe NLP domain [33], [34] or cover generic attention-basedapproaches [27], [33]. By focusing on the newly emergingarea of visual Transformers , we comprehensively organizethe recent approaches according to the intrinsic features ofself-attention and the investigated task. We first provide anintroduction to the salient concepts underlying Transformernetworks and then elaborate on the specifics of recent visiontransformers. Where ever possible, we draw parallels be-tween the Transformers used in the NLP domain [1] and theones developed for vision problems to flash major noveltiesand interesting domain-specific insights.

8 Recent approachesshow that convolution operations can be fully replacedwith attention-based transformer modules and have alsobeen used jointly in a single design to encourage symbiosisbetween the two complementary set of operations. This sur-vey finally details open research questions with an outlooktowards the possible future FOUNDATIONST here exist two key ideas that have contributed towardsthe development of conventional transformer models. (a)The first one isself-attention, which allows capturing long-term dependencies between sequence elements as com-pared to conventional recurrent models that find it chal-lenging to encode such relationships. (b) The second keyidea is that ofpre-training1on a large (un)labelled corpus ina (self)supervised manner, and subsequently fine-tuning tothe target task with a small labeled dataset [3], [7], [38]. Be-low, we provide a brief tutorial on these two ideas (Sec. ), along with a summary of seminal Transformer1.

9 Several recent Vision Transformers demonstrate that the modelcan be learned end-to-end on ImageNet-1K without any dedicated pre-training phase [35] [37]. However, the performance generally remainslower than the pre-trained 2:An example self-attention block used in the visiondomain [39]. Given the input sequence of image features, thetriplet of (key, query, value) is calculated followed by attentioncalculation and applying it to reweight the values. A singlehead is shown here and an output projection (W) is finallyapplied to obtain output features with the same dimension asthe input. Figure adapted from [39].networks (Sec. and ) where these ideas have beenapplied. This background will help us better understandthe forthcoming Transformer based models used in thecomputer vision domain (Sec. 3). Self-Attention in TransformersGiven a sequence of items, self-attention estimates the rel-evance of one item to other items ( , which words arelikely to come together in a sentence).

10 The self-attentionmechanism is an integral component of Transformers , whichexplicitly models the interactions between all entities of asequence for structured prediction tasks. Basically, a self-attention layer updates each component of a sequence byaggregating global information from the complete inputsequence. Lets denote a sequence ofnentities (x1,x2, xn)byX Rn d, wheredis the embedding dimension to rep-resent each entity. The goal of self-attention is to capture theinteraction amongst allnentities by encoding each entityin terms of the global contextual information. This is doneby defining three learnable weight matrices to transformQueries (WQ Rd dq), Keys (WK Rd dk) and Values(WV Rd dv), wheredq=dk. The input sequenceXisfirst projected onto these weight matrices to getQ=XWQ,K=XWKandV=XWV. The outputZ Rn dvof theself attention layer is,Z= softmax (QKT dq) 3:Architecture of the Transformer Model[1]. The model was first developed for the language translation task where an inputsequence in one language is required to be converted to the output sequence in another language.