TransReID: Transformer-Based Object Re-Identification

1 TransReID: Transformer-Based Object Re-IdentificationShuting He1,2*Hao Luo2 Pichao Wang2 Fan Wang2 Hao Li2 Wei Jiang1 1 Zhejiang University2 Alibaba robust feature representation is one of thekey challenges in Object Re-Identification (ReID). Althoughconvolution neural network (CNN)-based methods haveachieved great success, they only process one localneighborhood at a time and suffer from information loss ondetails caused by convolution and downsampling operators( and strided convolution).To overcomethese limitations, we propose a pure transformer-basedobject ReID framework named TransReID. Specifically,we first encode an image as a sequence of patchesand build a Transformer-Based strong baseline with afew critical improvements, which achieves competitiveresults on several ReID benchmarks with CNN-basedmethods.

2 To further enhance the robust feature learningin the context of transformers, two novel modulesare carefully designed.(i) The jigsaw patch module(JPM) is proposed to rearrange the patch embeddingsvia shift and patch shuffle operations which generatesrobust features with improved discrimination ability andmore diversified coverage.(ii) The side informationembeddings (SIE) is introduced to mitigate feature biastowards camera/view variations by plugging in learnableembeddings to incorporate these non-visual clues. To thebest of our knowledge, this is the first work to adopt apure transformer for ReID research. Experimental resultsof TransReID are superior promising, which achieve state-of-the-art performance on both person and vehicle ReIDbenchmarks.

3 Code is available IntroductionObject Re-Identification (ReID) aims to associate aparticular Object across different scenes and camera views,such as in the applications of person ReID and vehicleReID. Extracting robust and discriminative features is acrucial component of ReID, and has been dominated by*This work was done when Shuting He was intern at Alibabasupervised by Hao Luo and Pichao Wang. Corresponding author(a)(b)(c)(d)(a)(b)(c)(d)(a)(b)(c)( d)Figure 1: Grad-CAM [34] visualization of attention maps: (a)Original images, (b) CNN-based methods, (c) CNN+attentionmethods, (d) Transformer-Based methods which captures globalcontext information and more discriminative :16 CNN-basedmethodsID:443 Transformer-basedmethodsCNN-basedmethods Transformer-basedmethodsFigure 2: Visualization of output feature maps for 2 hard sampleswith similar methods retainbackpack details on output feature maps in contrast to CNN-basedmethods, as noted in red boxes.

4 For better visualization, inputimages are scaled to size1024 methods for a long time [19, 37, 36, 44, 42,5, 12, 13, 53, 15].By reviewing CNN-based methods, we find twoimportant issues which are not well addressed in the fieldof Object ReID. (1) Exploiting therich structural patternsin a global scopeis crucial for Object ReID [54]. However,CNN-based methods mainly focus on small discriminativeregions due to a Gaussian distribution of effective receptivefields [29]. Recently, attention modules [54, 6, 3, 21, 1]have been introduced to explore long-range dependencies[45], but most of them are embedded in the deep layers anddo not solve the principle problem of CNN. Thus, attention-based methods still prefer large continuous areas and arehard to extract multiple diversified discriminative parts (seeFigure 1).

5 (2) Fine-grained features withdetail informationare also important. However, the downsampling operators( pooling and strided convolution) of CNN reducespatial resolution of output feature maps, which greatlyaffect the discrimination ability to distinguish objects withsimilar appearances [37, 27]. As shown in Figure 2, thedetails of the backpack are lost in CNN-based feature maps,making it difficult to differentiate the two , Vision Transformer (ViT) [8] and Data-efficient image Transformers (DeiT) [40] have shownthat pure transformers can be as effective as CNN-based methods on feature extraction for image the introduction of multi-head attention modulesand the removal of convolution and downsamplingoperators, Transformer-Based models are suitable to solvethe aforementioned problems in CNN-based ReID forthe following reasons.

6 (1) The multi-head self-attentioncaptures long range dependencies and drives the modelto attend diverse human-body parts than CNN models( thighs, shoulders, waist in Figure 1). (2) Withoutdownsampling operators, transformer can keep moredetailed information. For example, one can observe thatthe difference on feature maps around backpacks (markedby red boxes in Figure 2) can help the model easilydifferentiate the two people. These advantages motivate usto introduce pure transformers in the Object its great advantages as discussed above,transformers still need to be designed specifically forobject ReID to tackle the unique challenges, such asthe large variations ( occlusions, diversity of poses,camera perspective) in images.

7 Substantial efforts havebeen devoted to alleviating this challenge in CNN-basedmethods. Among them, local part features [37, 44, 20, 48,28] and side information (such as cameras and viewpoints)[7, 61, 35, 30], have been proven to be essential andeffective to enhance the feature aggregated features makes it robust againstocclusions and misalignments [49]. However, extendingthe rigid stripe part methods from CNN-based methods topure Transformer-Based methods may damage long-rangedependencies due to global sequences splitting into severalisolated subsequences. In addition, taking side informationinto consideration, such as camera and viewpoint-specificinformation, an invariant feature space can be constructedto diminish bias brought by side information , the complex designs for side information built onCNN, if directly applied to transformers, cannot make fulluse of the inherent encoding capabilities of a result, specific designed modules are inevitable andessential for a pure transformer to successfully handle , we propose a new Object ReID frameworkdubbed TransReID to learn robust feature , by making several critical adaptations, we constructa strong baseline framework based on a pure , in order to expand long-range dependenciesand enhance feature robustness.

8 We propose ajigsawpatches module(JPM) by rearranging the patch embeddingsvia shift and shuffle operations and re-grouping them forfurther feature learning. The JPM is employed on the lastlayer of the model to extract robust features in parallelwith the global branch which does not include this specialoperation. Hence, the network tends to extract perturbation-invariant and robust features with global context. Thirdly,to further enhance the learning of robust features, asideinformation embedding(SIE) is ofthe special and complex designs in CNN-based methodsfor utilizing these non-visual clues, we propose a unifiedframework that effectively incorporates non-visual cluesthrough learnable embeddings to alleviate the data biasbrought by cameras or viewpoints.

9 Taking cameras forexample, the proposed SIE helps address the vast pairwisesimilarity discrepancy between inter-camera and intra-camera matching (see Figure 6). SIE can also be easilyextended to include any non-visual clues other than the oneswe have our best knowledge, we are the first to investigate theapplication of pure transformers in the field of Object contributions of the paper are summarised: We propose a strong baseline that exploits the puretransformer for ReID tasks for the first time andachieve comparable performance with CNN-basedframeworks. Wedesignajigsawpatchesmodule(JPM),consis ting of shift and patch shuffle operation,which facilitates perturbation-invariant and robustfeature representation of objects.

10 We introduce aside information embeddings(SIE) thatencodes side information by learnable embeddings,and is shown to effectively mitigate the feature bias. The final framework TransReID achieves state-of-the-art performance on both person and vehicleReIDbenchmarksincludingMSMT17[46] ,Market-1501[55], DukeMTMC-reID[33], Occluded-Duke[31], VeRi-776[25] and VehicleID[24].2. Related Object ReIDThe studies of Object ReID have been mainly focused onperson ReID and vehicle ReID, with most state-of-the-artmethods based on the CNN structure. A popular pipelinefor Object ReID is to design suitable loss functions to train aCNN backbone ( ResNet [11]), which is used to extractfeatures of images. The cross-entropy loss (ID loss) [56]and triplet loss [23] are most widely used in the deep al.