SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ...

1 SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC. Nicolai Wojke , Alex Bewley , Dietrich Paulus . University of Koblenz-Landau , Queensland University of Technology . ABSTRACT. [ ] 21 Mar 2017. SIMPLE ONLINE and REALTIME TRACKING (SORT) is a pragmatic approach to multiple object TRACKING with a focus on SIMPLE , effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer peri- ods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a large- scale person re-identification dataset.

2 During ONLINE appli- cation, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experi- Fig. 1: Exemplary output of our method on the MOT chal- mental evaluation shows that our extensions reduce the num- lenge dataset [15] in a common TRACKING situation with fre- ber of identity switches by 45%, achieving overall competi- quent occlusion. tive performance at high frame rates. Index Terms Computer Vision, Multiple Object Track- much simpler framework that performs Kalman filtering in ing, Data Association image space and frame-by-frame data association using the Hungarian method with an association metric that measures 1.

3 INTRODUCTION bounding box overlap. This SIMPLE approach achieves favor- able performance at high frame rates. On the MOT challenge Due to recent progress in object detection, TRACKING -by- dataset [13], SORT with a state-of-the-art people detector [14]. detection has become the leading paradigm in multiple object ranks on average higher than MHT on standard detections. TRACKING . Within this paradigm, object trajectories are usually This not only underlines the influence of object detector per- found in a global optimization problem that processes entire formance on overall TRACKING results, but is also an important video batches at once. For example, flow network formula- insight from a practitioners point of view.

4 Tions [1, 2, 3] and probabilistic graphical models [4, 5, 6, 7] While achieving overall good performance in terms of have become popular frameworks of this type. However, TRACKING precision and accuracy, SORT returns a relatively due to batch processing, these methods are not applicable high number of identity switches. This is, because the em- in ONLINE scenarios where a target identity must be available ployed association metric is only accurate when state esti- at each time step. More traditional methods are Multiple mation uncertainty is low. Therefore, SORT has a deficiency Hypothesis TRACKING (MHT) [8] and the Joint Probabilistic in TRACKING through occlusions as they typically appear in Data Association Filter (JPDAF) [9].

5 These methods perform frontal-view camera scenes. We overcome this issue by re- data association on a frame-by-frame basis. In the JPDAF, placing the association metric with a more informed metric a single state hypothesis is generated by weighting individ- that combines motion and appearance information. In par- ual measurements by their association likelihoods. In MHT, ticular, we apply a convolutional neural network (CNN) that all possible hypotheses are tracked, but pruning schemes has been trained to discriminate pedestrians on a large-scale must be applied for computational tractability. Both meth- person re-identification dataset. Through integration of this ods have recently been revisited in a TRACKING -by-detection network we increase robustness against misses and occlusions scenario [10, 11] and shown promising results.

6 However, the while keeping the system easy to implement, efficient, and performance of these methods comes at increased computa- applicable to ONLINE scenarios. Our code and a pre-trained tional and implementation complexity. CNN model are made publicly available to facilitate research SIMPLE ONLINE and REALTIME TRACKING (SORT) [12] is a experimentation and practical application development. 2. SORT WITH DEEP ASSOCIATION METRIC standard deviations the detection is away from the mean track location. Further, using this metric it is possible to exclude We adopt a conventional single hypothesis TRACKING methodol- unlikely associations by thresholding the Mahalanobis dis- ogy with recursive Kalman filtering and frame-by-frame data tance at a 95% confidence interval computed from the inverse association.

7 In the following section we describe the core 2 distribution. We denote this decision with an indicator components of this system in greater detail. bi,j = 1[d(1) (i, j) t(1) ]. (1). (2). Track Handling and State Estimation that evaluates to 1 if the association between the i-th track and j-th detection is admissible. For our four dimensional The track handling and Kalman filtering framework is mostly measurement space the corresponding Mahalanobis threshold identical to the original formulation in [12]. We assume is t(1) = a very general TRACKING scenario where the camera is un- While the Mahalanobis distance is a suitable association calibrated and where we have no ego-motion information metric when motion uncertainty is low, in our image-space available.

8 While these circumstances pose a challenge to problem formulation the predicted state distribution obtained the filtering framework, it is the most common setup con- from the Kalman filtering framework provides only a rough sidered in recent multiple object TRACKING benchmarks [15]. estimate of the object location. In particular, unaccounted Therefore, our TRACKING scenario is defined on the eight di- camera motion can introduce rapid displacements in the im- mensional state space (u, v, , h, x , y , , h ) that contains the age plane, making the Mahalanobis distance a rather unin- bounding box center position (u, v), aspect ratio , height formed metric for TRACKING through occlusions.

9 Therefore, we h, and their respective velocities in image coordinates. We integrate a second metric into the assignment problem. For use a standard Kalman filter with constant velocity motion each bounding box detection dj we compute an appearance and linear observation model, where we take the bounding descriptor r j with kr j k = 1. Further, we keep a gallery Rk =. coordinates (u, v, , h) as direct observations of the object (i). state. {r k }L k=1 of the last Lk = 100 associated appearance de- k For each track k we count the number of frames since the scriptors for each track k. Then, our second metric measures last successful measurement association ak.

10 This counter is the smallest cosine distance between the i-th track and j-th incremented during Kalman filter prediction and reset to 0 detection in appearance space: when the track has been associated with a measurement. (i) (i). d(2) (i, j) = min{1 r j T r k | r k Ri }. (3). Tracks that exceed a predefined maximum age Amax are con- sidered to have left the scene and are deleted from the track Again, we introduce a binary variable to indicate if an associ- set. New track hypotheses are initiated for each detection that ation is admissible according to this metric cannot be associated to an existing track. These new tracks bi,j = 1[d(2) (i, j) t(2) ]. (2).