Abstract - arXiv

1 Transformer Meets Tracker:Exploiting Temporal Context for Robust Visual TrackingNing Wang1 Wengang Zhou1,2 Jie Wang1,2 Houqiang Li1,21 CAS Key Laboratory of GIPAS, EEIS Department, University of Science and Technology of China (USTC)2 Institute of Artificial Intelligence, Hefei Comprehensive National Science video object tracking, there exist rich temporal con-texts among successive frames, which have been largelyoverlooked in existing trackers. In this work, we bridge theindividual video frames and explore the temporal contextsacross them via a transformer architecture for robust objecttracking. Different from classic usage of the transformer innatural language processing tasks, we separate its encoderand decoder into two parallel branches and carefully designthem within the Siamese-like tracking pipelines.

2 The trans-former encoder promotes the target templates via attention-based feature reinforcement, which benefits the high-qualitytracking model generation. The transformer decoder prop-agates the tracking cues from previous templates to the cur-rent frame, which facilitates the object searching transformer-assisted tracking framework is neat andtrained in an end-to-end manner. With the proposed trans-former, a simple Siamese matching approach is able to out-perform the current top-performing trackers. By combin-ing our transformer with the recent discriminative track-ing pipeline, our method sets several new state-of-the-artrecords on prevalent tracking IntroductionVisual object tracking is a basic task in computer the recent progress, it remains a challenging taskdue to factors such as occlusion, deformation, and appear-ance changes.

3 With the temporal error accumulation, thesechallenges are further amplified in the online is well recognized that the rich temporal informationin the video flow is of vital importance for visual track-ing. However, most tracking paradigms [29, 28, 49] handlethis task by per-frame object detection, where the tempo-*Corresponding Author: Wengang Zhou and Houqiang Li. Source code, pretrained model, and raw tracking results are SetSearch PatchTransformerResponseBackboneTracking ModelWeight SharingTransformerEncoderFigure 1. An overview of our transformer-assisted tracking frame-work. The transformer encoder and decoder are assigned to twoparallel branches in a Siamese-like tracking pipeline.

4 Thanks tothe encoder-decoder structure, isolated frames are tightly bridgedto convey rich temporal information in the video relationships among successive frames have been largelyoverlooked. Take the popular Siamese tracker as an exam-ple, only the initial target is considered for template match-ing [1, 45, 19, 29]. The merely used temporal informationis the motion prior ( ,cosine window) by assuming thetarget moves smoothly, which is widely adopted in visualtrackers. In other tracking frameworks with update mech-anisms [20, 40, 8, 60, 62, 3], previous prediction resultsare collected to incrementally update the tracking the historical frames considered in the above ap-proaches, the video frames are still considered as indepen-dent counterparts without mutual reasoning.

5 In real-worldvideos, some frames inevitably contain noisy contents suchas occluded or blurred objects. These imperfect frames willhurt the model update when serving as the templates andwill challenge the tracking process when performing as thesearch frames. Therefore, it is a non-trivial issue to conveyrich information across temporal frames to mutually rein-force them. We argue that the video frames should not betreated in isolation and the performance potential is largelyrestricted due to the overlook of frame-wise bridge the isolated video frames and convey the richtemporal cues across them, in this work, we introduce thetransformer architecture [47] to the visual tracking commu-nity.

6 Different from the traditional usage of the transformer1 [ ] 24 Mar 2021 Template FeaturesTransformerEncoderTransformerDec oderTemplate PatchesSearch PatchMask PropagationFeature PropagationEncoded Features Masks Transformed Feature and MaskSearch FeatureTemplate BranchSearch BranchFeature and MaskPropagationDecoded Feature Figure 2. Top: the transformer encoder receives multiple templatefeatures to mutually aggregate representations. Bottom: the trans-former decoder propagates the template features and their assignedmasks to the search patch feature for representation language modeling and machine translation [47, 12], weleverage it to handle thecontext propagation in the tempo-ral domain.

7 By carefully modifying the classic transformerarchitecture, we show that its transformation characteristicnaturally fits the tracking scenario. Its core component, ,attention mechanism [47, 57], is ready to establish the pixel-wise correspondence across frames and freely convey vari-ous signals in the temporal , most tracking methods [1, 46, 29, 43, 7, 3] canbe formulated into a Siamese-like framework, where the topbranch learns a tracking model using template features, andthe bottom branch classifies the current search patch. Asshown in Figure 1, we separate the transformer encoder anddecoder into two branches within such a general Siamese-like structure.

8 In the top branch, a set of template patchesare fed to the transformer encoder to generate high-qualityencoded features. In the bottom branch, the search featureas well as the previous template contents are fed to the trans-former decoder, where the search patch retrieves and aggre-gates informative target cues ( ,spatial masks and targetfeatures) from history templates to reinforce proposed transformer facilitates visual tracking via: Transformer enables individual templatefeatures to mutually reinforce to acquire more compacttarget representations, as shown in Figure 2. These en-coded high-quality features further benefit the trackingmodel generation.

9 Transformer conveys valuable temporalinformation across frames. As shown in Figure 2, ourdecoder simultaneously transfers features and spatialmasks. Propagating the features from previous framesto the current patch smooths the appearance changesand remedies the context noises while transforming thespatial attentions highlights the potential object loca-tion. These manifold target representations and spatialcues make the object search much , we track the target in the decoded search patch. Toverify the generalization of our designed transformer, weintegrate it into two popular tracking frameworks includ-ing a Siamese formulation [1] and a discriminative corre-lation filter (DCF) based tracking paradigm [3].

10 With ourdesigned transformer, a simple Siamese matching pipelineis able to outperform the current top-performing combining with the recent discriminative approach [3],our transformer-assisted tracker shows outstanding resultson seven prevalent tracking benchmarks including LaSOT[13], TrackingNet [39], GOT-10k [23], UAV123 [37], NfS[24], OTB-2015 [58], and VOT2018 [26] and sets severalnew state-of-the-art summary, we make three-fold contributions: We present a neat and novel transformer-assisted track-ing framework. To our best knowledge, this is the firstattempt to involve the transformer in visual tracking. We simultaneously consider the feature and attentiontransformations to better explore the potential of thetransformer.