1 Multiple Object Tracking: A Literature Review - …

1 1 Multiple Object tracking : A Literature ReviewWenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Xiaowei Zhao and Tae-Kyun KimAbstract Multiple Object tracking (MOT) is an important computer vision problem which has gained increasing attention due toits academic and commercial potential. Although different kinds of approaches have been proposed to tackle this problem, it stillremains challenging due to factors like abrupt appearance changes and severe Object occlusions. In this work, we contribute thefirst comprehensive and most recent Review on this problem. We inspect the recent advances in various aspects and propose someinteresting directions for future research.

2 To the best of our knowledge, there has not been any extensive Review on this topic in thecommunity. We endeavor to provide a thorough Review on the development of this problem in recent decades. The main contributionsof this Review are fourfold: 1) Key aspects in a Multiple Object tracking system, including formulation, categorization, key principles,evaluation of an MOT are discussed. 2) Instead of enumerating individual works, we discuss existing approaches according to variousaspects, in each of which methods are divided into different groups and each group is discussed in detail for the principles, advancesand drawbacks. 3) We examine experiments of existing publications and summarize results on popular datasets to provide quantitativecomparisons.

3 We also point to some interesting discoveries by analyzing these results. 4) We provide a discussion about issues of MOTresearch, as well as some interesting directions which could possibly become potential research effort in the INTRODUCTIONM ultiple Object tracking (MOT), or Multiple TargetTracking (MTT), plays an important role in computer vi-sion. The task of MOT is largely partitioned to locatingmultiple objects, maintaining their identities, and yieldingtheir individual trajectories given an input video. Objects totrack can be, for example, pedestrians on the street [1], [2],vehicles in the road [3], [4], sport players on the court [5],[6], [7], or groups of animals (birds [8], bats [9], ants [10],fish [11], [12], [13], cells [14], [15],etc.

4 Multiple objects could also be viewed as different parts of a single Object [16]. In this Review , we mainly focus on the research onpedestrian tracking . The underlying reasons for this spec-ification are threefold. First, compared to other commonobjects in our environment, pedestrians are typical non-rigid objects, which is an ideal example to study the MOTproblem. Second, videos of pedestrians arise in a huge num-ber of practical applications, which further results in greatcommercial potential. Third, according to all data collectedfor this Review , at least70%of current MOT research effortsare devoted to a mid-level task in computer vision, Multiple objecttracking grounds high-level tasks such as pose estimation[17], action recognition [18], and behavior analysis [19].

5 Ithas numerous practical applications, such as visual surveil-lance [20], human computer interaction [21] and virtualreality [22]. These practical requirements have sparked enor-mous interest in this topic. Compared with Single ObjectTracking (SOT), which primarily focuses on designing so-phisticated appearance models and/or motion models todeal with challenging factors such as scale changes, out-of-plane rotations and illumination variations, Multiple ob-ject tracking additionally requires two tasks to be solved:determining the number of objects, which typically variesover time, and maintaining their identities. Apart fromthe common challenges in both SOT and MOT, furtherkey issues that complicate MOT include among others:1) frequent occlusions, 2) initialization and termination oftracks, 3) similar appearance, and 4) interactions amongmultiple objects.

6 In order to deal with all these issues, a widerange of solutions have been proposed in the past solutions concentrate on different aspects of an MOTsystem, making it difficult for MOT researchers, especiallynewcomers, to gain a comprehensive understanding of thisproblem. Therefore, in this work we provide a Review todiscuss the various aspects of the Multiple Object Differences from Other Related ReviewsTo the best of our knowledge, there has not been anycomprehensive Literature Review on the topic of multipleobject tracking . However, there have been some other re-views related to Multiple Object tracking , which are listed inTable 1. We group these surveys into three sets and highlightthe differences from ours as follows.

7 The first set [19], [20], [21], [23], [24] discusses track-ing as an individual part while this work specificallydiscusses various aspects of MOT. For example, ob-ject tracking is discussed as a step in the procedureof high-level tasks such as crowd modeling [19], [23],[24]. Similarly, in [21] and [20], Object tracking is re-viewed as a part of a system for behavior recognition[21] or video surveillance [20]. The second set [25], [26], [27], [28] is dedicated togeneral visual tracking techniques [25], [26], [27] orsome special issues such as appearance models invisual tracking [28]. Their reviewing scope is widerthan ours; ours on the contrary is more comprehen-sive and focused on Multiple Object tracking .

8 The third set [29], [30] introduces and discussesbenchmarks on general visual tracking [29] and onspecific Multiple Object tracking [30]. Their attentionis laid on experimental studies rather than [ ] 22 May 20172 TABLE 1: A summary of other Literature reviewsReferenceTopicYearZhanet al. [23]Crowd Analysis2008 Huet al. [19] Object Motion and Behaviors2004 Kimet al. [24]Intelligent Visual Surveillance2010 Candamoet al. [21]Behavior Recognition in Transit Scenes 2010 Xiaogang Wang [20] Multi-Camera Video Surveillance2013 Forsythet al. [25]Human Motion Analysis2006 Kevin Cannons [26] Visual Tracking2008 Yilmazet al. [27] Object Visual Tracking2006 Liet al. [28]Appearance Models in Object tracking 2013 Wuet al.

9 [29]Visual tracking Benchmark2013 Leal-Taix eet al. [30] MOT ContributionsWe provide the first comprehensive Review on the MOTproblem to the computer vision community, which webelieve is helpful to understand this problem, its mainchallenges, pitfalls, and the state of the art. The main contri-butions of this Review are summarized as follows: We derive a unified formulation of the MOT prob-lem which consolidates most of the existing MOTmethods (Section ), and two different ways tocategorize MOT methods (Section ). We investigate different key components involved inan MOT system, each of which is further divided intodifferent aspects and discussed in detail regarding itsprinciples, advances, and drawbacks (Section 3).

10 Experimental results on popular datasets regardingdifferent approaches are presented, which makesfuture experimental comparison convenient. By in-vestigating the provided results, some interestingobservations and findings are revealed (Section 4). By summarizing the MOT Review , we unveil existingissues of MOT research. Furthermore, open problemsare discussed to identify potential future researchdirections (Section 5).Note that this work is mainly dedicated to reviewing recentliterature on the advances in Multiple Object tracking . Asmentioned above, we also present experimental results onpublicly available datasets excepted from existing publica-tions to provide a quantitative view on the state-of-the-artMOT methods.