Rethinking Semantic Segmentation From a Sequence-to ...

1 Rethinking Semantic Segmentation from a Sequence-to -Sequence Perspectivewith TransformersSixiao Zheng1*Jiachen Lu1 Hengshuang Zhao2 Xiatian Zhu3 Zekun Luo4 Yabiao Wang4 Yanwei Fu1 Jianfeng Feng1 Tao Xiang3, 5 Philip Torr2Li Zhang1 1 Fudan University2 University of Oxford3 University of Surrey4 Tencent Youtu Lab5 Facebook recent Semantic Segmentation methods adopta fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reducesthe spatial resolution and learns more abstract/semanticvisual concepts with larger receptive fields.

2 Since contextmodeling is critical for Segmentation , the latest efforts havebeen focused on increasing the receptive field, through ei-ther dilated/atrous convolutions or inserting attention mod-ules. However, the encoder-decoder based FCN architec-ture remains unchanged. In this paper, we aim to providean alternative perspective by treating Semantic segmenta-tion as a Sequence-to -sequence prediction task. Specifically,we deploy a pure transformer ( , without convolution andresolution reduction) to encode an image as a sequence ofpatches.

3 With the global context modeled in every layer ofthe transformer, this encoder can be combined with a simpledecoder to provide a powerful Segmentation model, termedSEgmentation TRansformer (SETR). Extensive experimentsshow that SETR achieves new state of the art on ADE20K( mIoU), Pascal Context ( mIoU) and com-petitive results on Cityscapes. Particularly, we achieve thefirstposition in the highly competitive ADE20K test serverleaderboard on the day of IntroductionSince the seminal work of [35], existing Semantic seg-mentation models have been dominated by those based onfully convolutional network (FCN).

4 A standard FCN seg-*Work done while Sixiao Zheng was interning at Tencent Youtu Lab. Li Zhang is the corresponding author withSchool of Data Science, Fudan University. Yanwei Fu is with the Schoolof Data Science, MOE Frontiers Center for Brain Science, and ShanghaiKey Lab of Intelligent Information Processing, Fudan University. JianfengFeng is with the Institute of Science and Technology for Brain-InspiredIntelligence, Fudan model has an encoder-decoder architecture: theencoderis for feature representation learning, while thede-coderfor pixel-level classification of the feature representa-tions yielded by the encoder.

5 Among the two, feature rep-resentation learning ( , the encoder) is arguably the mostimportant model component [7,27,55,58]. The encoder,like most other CNNs designed for image understanding,consists of stacked convolution layers. Due to concernson computational cost, the resolution of feature maps is re-duced progressively, and the encoder is hence able to learnmore abstract/ Semantic visual concepts with a gradually in-creased receptive field. Such a design is popular due to twofavorable merits, namely translation equivariance and local-ity.

6 The former respects well the nature of imaging pro-cess [56] which underpins the model generalization abilityto unseen image data. Whereas the latter controls the modelcomplexity by sharing parameters across space. However, italso raises a fundamental limitation that learning long-rangedependency information, critical for Semantic segmentationin unconstrained scene images [1,48], becomes challengingdue to still limited receptive overcome this aforementioned limitation, a numberof approaches have been introduced recently.

7 One approachis to directly manipulate the convolution operation. This in-cludes large kernel sizes [39], atrous convolutions [7,21],and image/feature pyramids [58]. The other approach is tointegrate attention modules into the FCN architecture. Sucha module aims to modelglobalinteractions of all pixels inthe feature map [47]. When applied to Semantic segmenta-tion [24,28], a common design is to combine the attentionmodule to the FCN architecture with attention layers sittingon the top. Taking either approach, the standard encoder-decoder FCN model architecture remains unchanged.

8 Morerecently, attempts have been made to get rid of convolutionsaltogether and deploy attention-alone models [46] , even without convolution, they do not change thenature of the FCN model structure: an encoder downsam-6881ples the spatial resolution of the input, developing lower-resolution feature mappings useful for discriminating se-mantic classes, and the decoder upsamples the feature rep-resentations into a full-resolution Segmentation this paper, we aim to provide a Rethinking to the se-mantic Segmentation model design and contribute an alter-native.

9 In particular, we propose to replace the stacked con-volution layers based encoder with gradually reduced spa-tial resolution with a pure transformer [44], resulting ina new Segmentation model termedSEgmentation TRans-former(SETR). This transformer-alone encoder treats aninput image as a sequence ofimage patchesrepresentedby learned patch embedding, and transforms the sequencewith global self-attention modeling for discriminative fea-ture representation learning. Concretely, we first decom-pose an image into a grid of fixed-sized patches, forming asequence of patches.

10 With a linear embedding layer appliedto the flattened pixel vectors of every patch, we then obtaina sequence of feature embedding vectors as the input to atransformer. Given the learned features from the encodertransformer, a decoder is then used to recover the originalimage resolution. Crucially there isnodownsampling inspatial resolution but global context modeling at every layerof the encoder transformer, thus offering a completely newperspective to the Semantic Segmentation pure transformer design is inspired by its tremen-dous success in natural language processing (NLP) [13,44].