Track To Detect and Segment: An Online Multi-Object Tracker

1 Track to Detect and Segment: An Online Multi-Object TrackerJialian Wu1, Jiale Cao2, Liangchen Song1, Yu Wang3, Ming Yang3, Junsong Yuan11 SUNY Buffalo2 TJU3 Horizon RoboticsAbstractMost Online Multi-Object trackers perform object detec-tion stand-alone in a neural net without any input fromtracking. In this paper, we present a new Online jointdetection and tracking model, TraDeS ( Track to DEtectand Segment), exploiting tracking clues to assist detectionend-to-end. TraDeS infers object tracking offset by a costvolume, which is used to propagate previous object fea-tures for improving current object detection and segmen-tation. Effectiveness and superiority of TraDeS are shownon 4 datasets, including MOT (2D tracking), nuScenes (3 Dtracking), MOTS and Youtube-VIS (instance segmentationtracking). Project page: IntroductionAdvanced Online Multi-Object tracking methods followtwo major paradigms: tracking-by-detection [5,38,27,52,30,49] and joint detection and tracking [26,63,1,29,45,25,43,44].

2 The tracking-by-detection (TBD) paradigm treatsdetection and tracking as two independent tasks ( (a)).It usually applies an off-the-shelf object detector to producedetections and employs another separate network for dataassociation. TheTBDsystem is inefficient and not optimizedend-to-end due to the two-stage processing. To address thisproblem, recent solutions favor a joint detection and tracking(JDT) paradigm that simultaneously performs detection andtracking in a single forward-pass ( (b)).TheJDTmethods, however, are confronted with two is-sues:(i)Although in mostJDTworks [29,45,25,50] thebackbone network is shared, detection is usually performedstandalone without exploring tracking cues. We argue thatdetection is the cornerstone for a stable and consistent Track -let, and in turn tracking cues shall help detection, especiallyin tough scenarios like partial occlusion and motion blur.(ii)As studied by [9] and our experiment ( ), common re-ID tracking loss [45,25,32,51] is not that compatible withdetection loss in jointly training a single backbone network,which could even hurt detection performance to some image(a) Tracking-by-DetectionDetection and Tracking(c) TraDeS (Ours)Input imageInput imageLearnable NetworksForward(b) Recent Joint Data AssociationmissedmissedData AssociationPrevious FeaturesData AssociationdetectedD: Detection LayerT:Tracking-related LayerRe-ID EmbeddingsTracking OffsetsDetected ObjectsTracked ObjectsDTbackboneEnhancebackbonebackbone backboneDTTDF igure of different Online MOT follows the joint detection and tracking (JDT) from mostJDTmethods, the proposed TraDeS trackerdeeply couples tracking and detection within an end-to-end and uni-fied framework, where the motion clue from tracking is exploitedto enhance detection or segmentation (omitted in the figure).

3 The reason is that re-ID focuses on intra-class variance, butdetection aims to enlarge inter-class difference and minimizeintra-class this paper, we propose a new Online joint detection andtracking model, coined as TraDeS ( Track to Detect andSegment). In TraDeS, each point on the feature map repre-sents either an object center or a background region, similaras in CenterNet [64]. TraDeS addresses the above two is-sues by tightly incorporating tracking into detection as wellas a dedicatedly designed re-ID learning scheme. Specifi-cally, we propose a cost volume based association (CVA)module and a motion-guided feature warper (MFW) module,respectively. The CVA extracts point-wise re-ID embeddingfeatures by the backbone to construct a cost volume thatstores matching similarities between the embedding pairs intwo frames. Then, we infer the tracking offsets from the costvolume, which are the spatio-temporal displacements of allthe points, ,potential object centers, in two frames.

4 Thetracking offsets together with the embeddings are utilized toconduct a simple two-round long-term data association. Af-terwards, the MFW takes the tracking offsets as motion cuesto propagate object features from the previous frames to thecurrent one. Finally, the propagated feature and the currentfeature are aggregated to derive detection and the CVA module, the cost volume is employed to su-12352pervise the re-ID embedding, where different object classesand background regions are implicitly taken into is being said, our re-ID objective involves the inter-classvariance. This way not only learns an effective embeddingas common re-ID loss [45,25,32,51], but also is well com-patible with the detection loss and does not hurt detectionperformance as shown in Moreover, because thetracking offset is predicted based on appearance embeddingsimilarities, it can match an object with very large motionor in low frame rate as shown in , or even accuratelytrack objects in different datasets with unseen large motionas shown in Thus, the predicted tracking offset of anobject can serve as a robust motion clue to guide our featurepropagation in the MFW module.

5 The occluded and blurredobjects in the current frame may be legible in early frames,so the propagated features from previous frames may supportthe current feature to recover potentially missed objects byour MFW summary, we propose a novel Online multi-objecttracker, TraDeS, that deeply integrates tracking cues to assistdetection in an end-to-end framework and in return benefitstracking as shown in (c). TraDeS is a general Tracker ,which is readily extended to instance segmentation trackingby adding a simple instance segmentation branch. Exten-sive experiments are conducted on 4 datasets, , MOT,nuScenes, MOTS, and Youtube-VIS datasets, across 3 tasksincluding 2D object tracking, 3D object tracking, and in-stance segmentation tracking. TraDeS achieves state-of-the-art performance with an efficient inference time as shownin Additionally, thorough ablation studies are per-formed to demonstrate the effectiveness of our approach asshown in Related was dominated by thetracking-by-detection (TBD) paradigm over the pastyears [58,6,66,52,33,5,38,48,54].

6 Within this frame-work, an off-the-shelf object detector [31,16] is first appliedto generate detection boxes for each frame. Then, a separatere-ID model [1,49] is used to extract appearance features forthose detected boxes. To build tracklets, one simple solutionis to directly compute appearance and motion affinities with amotion model, ,Kalman filter, and then solve data associ-ation by a matching algorithm. Some other efforts [6,46,19]formulate data association as a graph optimization problemby treating each detection as a graph node. However,TBDmethods conduct detection and tracking separately, hence areusually computationally expensive. Instead, our approach in-tegrates tracking cues into detection and efficiently performsdetection and tracking in an end-to-end Detection and joint detectionand tracking (JDT) paradigm has raised increasing attentiondue to its efficient and unified framework. One commonway [63,45,25,1,62,61] is to build a tracking-relatedbranch upon an object detector to predict either object Track -ing offsets or re-ID embeddings for data association.

7 Alter-natively, transformer is exploited to match tracklets [36,26].CTracker [29] constructs tracklets by chaining paired boxesin every two frames. TubeTK [28] directly predicts a boxtube as a tracklet in an offline manner. MostJDTmethods,however, are confronted with two issues: First, detectionis still separately predicted without the help from , the re-ID loss has a different objective from thatof detection loss in joint training. In contrast, our TraDeStracker addresses these two problems by tightly incorporat-ing tracking cues into detection and designing a novel re-IDembedding learning Video Object video objectdetection, a few attempts [15,62] exploit tracking results toreweight the detection scores generated by an initial these works strive to help detection by tracking,they have two drawbacks: First, tracking is leveraged to helpdetection only at the post-processing stage. Detections arestill predicted by a standalone object detector, so detectionand tracking are separately optimized.

8 Thus, the final detec-tion scores may heavily rely on the tracking quality. Second,a hand-crafted reweighting scheme requires manual tune-upfor a specific detector and Tracker . Our approach differsfrom these post-processing methods because our detection islearned conditioned on tracking results, without a complexreweighting scheme. Therefore, detection tends to be cost volume technique has been success-fully applied in depth estimation [11,55,18] and opticalflow estimation [35,10,53] for associating pixels betweentwo frames. This motivates us to extend cost volume to amulti-object Tracker , which will be demonstrated to be ef-fective in learning re-ID embeddings and inferring trackingoffsets in this paper. Our approach may inspire future worksusing cost volume in tracking or PreliminariesThe proposed TraDeS is built upon the point-based ob-ject detector CenterNet [64]. CenterNet takes an imageI RH W 3as input and produces the base featuref= (I)via the backbone network ( ), wheref RHF WF 64,HF=H4, andWF=W4.

9 A set of head convolutionalbranches are then constructed onfto yield a class-wisecenter heatmapP RHF WF Nclsand task-specific pre-diction maps, such as 2D object size map and 3D object sizemap, the number of classes. CenterNet detectsobjects by their center points (peaks onP) and the corre-sponding task-specific predictions from the peak to [63], we build a baseline Tracker by adding12353h offsetInput0 Weight-sharingWeight-sharing Conv-BNConvConv-BN H W 3 2H W 3H W 3 Conv-BNConvConv-BN 1 WC[-20, 0, 20 .. 180, 200]0, , 0, ..[ -20, 0, 20 .. ] , ..softmaxCVAmax (ax is=0)max (ax is=1)HC WC HC WCCost Volume MapHC 1MP+RDCNHead Networks2D3 DMasktConv-BNConvConv-BN HC WC HC WCDCNt -1 Weight-sharingWeight-sharingMP+RMP+RCost Volume MapMatrixMultiplicationHadamardProductEl ement-wiseSumMP+RMax Pool& ReshapeDCND eformable Conv2D3 DMask2D detection3D detectionInstance segmentation2D3 DMaskHead Networks 1t- 2 Head Networks-190== Track i n g O ffseth offs e t 1 2 2 1 2 1 1 , , , ( , )( , ) -19w offs e t w offset 1 1 2 2 1 CVA.

10 1 1 HC WCMFWMFW , 1 PropagatedFeature 2 PropagatedFeatureDotProductTracking Offs e tFigure of may propagate features from multiple previous frames for object feature enhancement ( ,T >1),which is not shown in the above figure for extra head branch on CenterNet that predicts a trackingoffset mapOB RHF WF 2for data spatio-temporal displacements from all points attimetto the corresponding points at a previous timet .4. TraDeS TrackerOur Idea:Most previous joint detection and tracking meth-ods perform a standalone detection without explicit inputfrom tracking. In contrast, our aim is to integrate trackingcues into detection end-to-end, so as to improve detection fortough scenarios, which in return benefit tracking. To this end,we propose a Cost Volume based Association (CVA: )module for learning re-ID embeddings and deriving objectmotions, and a Motion-guided Feature Warper (MFW: )module for leveraging tracking cues from the CVA to propa-gate and enhance object Cost Volume based AssociationCost Volume:Given two base featuresftandft fromItandIt , we extract their re-ID embedding features by theembedding network ( ), ,et= (ft) RHF WF 128,where ( )consists of three convolution layers.