Multiple Object Tracking Performance Metrics and ...

1 Multiple Object Tracking Performance Metricsand Evaluation in a Smart Room EnvironmentKeni Bernardin, Alexander Elbs, Rainer StiefelhagenInstitut f ur Theoretische InformatikInteractive Systems LabUniversit at Karlsruhe, 76131 Karlsruhe, Tracking of Multiple persons in real worldenvironments is an active research field and several ap-proaches have been proposed, based on a variety of featuresand algorithms. Recently, there has been a growing interestin organizing systematic evaluations to compare the varioustechniques. Unfortunately, the lack of common Metrics formeasuring the Performance of Multiple Object trackers stillmakes it hard to compare their this work, we introduce two intuitive and general metricsto allow for objective comparison of tracker characteris-tics, focusing on their precision in estimating Object loca-tions, their accuracy in recognizing Object configurationsand their ability to consistently label objects over also present a novel system for Tracking Multiple users ina smart room environment using several cameras.

2 Based oncolor histogram Tracking of person regions and automaticinitialization using special Object system is used to demonstrate the expressiveness of theproposed Metrics through a sample Performance evaluationusing real test video sequences of people interacting in thesmart Introduction and Related WorkThe Tracking of Multiple persons in camera images is a veryactive research field with applications in many range from video surveillance, over automatic index-ing, to intelligent interactive environments. Especially inthe last case, a robust person Tracking module can serve asa poweful building block to support other techniques, suchas gesture recognizers, face identifiers, head pose estima-tors [10], scene analysis tools, etc. In the last few years,more and more approaches have been presented to tacklethe problems posed by unconstrained, natural environmentsand bring person trackers out of the laboratory environmentand into real world recent times, there has also been a growing interestin performing systematic evaluations of such Tracking toolswith common databases and Metrics .

3 Examples are theCHIL project, funded by the EU [17], the VACE projectin the [18], but also a growing number of workshops(PETS [19], EEMCV [20], etc). However, there is still nogeneral agreement on a principled evaluation procedure us-ing a common set of objective and intuitive Metrics for mea-suring the Performance of Multiple Object trackers. Becauseof this lack of Metrics , some researchers present their track-ing systems without any quantitative evaluation of their per-formance ( [1, 7, 15]). On the other hand, a multitude ofisolated measures were defined in individual contributionsto validate trackers using various features and algorithms(see [2, 3, 5, 11, 13]), but no common agreement on abest set of measures remedy this, this paper proposes a thorough proce-dure to detect all types of errors produced by a Multiple ob-ject Tracking system and introduces two novel Metrics , theMultiple Object Tracking Precision (M OT P), and the Mul-tiple Object Tracking accuracy (M OT A), that intuitivelyexpress a tracker s characteristics and could be used in gen-eral Performance the work that most closely relates to ours is thatof Smith et al.

4 In [4]. The authors also attempt to definean objective procedure and measures for Multiple objecttracker Performance . However, key differences to our con-tribution exist: In [4], the autors introduce a large number ofmetrics: 5 for measuring Object configuration errors, and 4for measuring inconsistencies in Object labeling over of the measures are defined in a dual way for trackersand for objects ( T/M O,F IT/F IO,T P/OP). Thiscould make it difficult to gain a clear and direct understand-ing of the tracker s overall Performance . Moreover, undercertain conditions, some of these measures can behave in anon-intuitive fashion (such as theCD, as the authors state,or theF PandF N, as we will demonstrate later). In com-parison, we introduce just 2 overall Performance measuresthat allow a clear and intuitive insight into the main trackercharacteristics: its precision in estimating Object positions,its ability to determine the number of objects and their con-figuration, and its skill at keeping consistent tracks overtime.

5 In addition, we offer an experimental validation ofthe presented theoretical framework by performing sampleevaluation runs on 2 variants of a Multiple person tracker,using real data recorded in a smart room environment. Ademonstration run on simulated data is also performed tobetter illustrate the expressiveness of the proposed system used in our evaluations is a 3D Multiple per-son tracker developed for use in our smart room. It initial-izes automatically using special person detectors, performscolor histogram Tracking of body parts on several cameraviews and intelligently fuses the 2D information to producea consistent set of 3D hypotheses. It is used as a proof ofconcept for the introducedM OT PandM OT Ametrics,which are used to measure its accuracy on datasets of vary-ing degrees of remainder of the paper is organized as follows:Section 2 presents the new Metrics , theM OT Pand theM OT Aand a detailed procedure for their 3 briefly introduces the developed multipersontracker which will be used in the evaluations.

6 In Section 4,the sample Performance measurements are shown and theusefulness of the Metrics is discussed. Finally, Section 5gives a summary and a Performance Metrics for MultipleObject TrackingTo help better understand the proposed evaluation Metrics ,we first explain what qualities we expect from an ideal mul-tiple Object tracker. It should at all points in time find thecorrect number of objects present and estimate the positionof each Object as precisely as possible (Note that propertiessuch as the size, contour, orientation or speed of objects arenot considered here). It should also keep consistent trackof each Object over time: Each Object should be assigneda unique track ID which stays constant throughout the se-quence (even after temporary occlusion, etc). This leadsto the following design criteria for Performance evaluationmetrics: They should allow to judge the tracker s precision indetermining exact Object locations.

7 They should reflect its ability to consistently track ob-ject configurations through time, to correctly traceobject trajectories, producing exactly one trajectoryper , we expect useful Metrics to have as few free parameters, adjustable thresholds,etc, as possible to help make evaluations straightfor-ward and keep results comparable. to be clear, easily understandable and behave accord-ing to human intuition, especially in the occurence ofmultiple errors of different types or of uneven reparti-tion of errors throughout the sequence. to be general enough to allow comparison of mosttypes of trackers (2D, 3D trackers, Object centroidtrackers or Object area trackers, etc). to be few in number and yet expressive, so they maybe used in large evaluations where many systemsare being on the above criteria, we propose a procedure forsystematic and objective evaluation of a tracker s charac-teristics.

8 Assuming that for every time frameta multipleobject tracker outputs a set of hypotheses{h1.. hm}for aset of visible objects{o1.. on}, the evaluation procedurecomprises the following steps:For each time framet, Establish the best possible correspondence betweenhypotheseshjand objectsoi For each found correspondence, compute the error inthe Object s position estimation. Accumulate all correspondence errors: Count all objects for which no hypothesis wasoutput as misses. Count all tracker hypotheses for which no realobject exists as false positives Count all occurences where the Tracking hypoth-esis for an Object changed compared to previousframes as mismatch errors. This could happen, , when two or more objects are swapped whenthey pass close to each other, or when an objecttrack is reinitialized with a different ID, after itwas previously lost because of , the Tracking Performance can be intuitively expressedin two numbers: The Tracking precision which expresseshow well exact positions of persons are estimated, andthe Tracking accuracy which shows how many mistakesthe tracker made in terms of misses, false positives, mis-matches, failures to recover tracks, etc.

9 These measures willbe explained in detail in the latter part of this Establishing Correspondences BetweenObjects and Tracker HypothesesAs explained above, the first step in evaluating the per-formance of a Multiple Object tracker is finding a contin-uous mapping between the sequence of Object hypotheses{h1.. hm}output by the tracker in each frame and the realobjects{o1.. on}. This is illustrated in Fig. 1. Naively,one would match the closest Object -hypothesis pairs andtreat all remaining objects as misses and all remaining hy-potheses as false positives. A few important points needto be considered, though, which make the procedure Valid CorrespondencesFirst of all, the correspondence between an objectoiand ahypothesishjshould not be made if their distancedisti,jexceeds a certain thresholdT. There is a certain concep-tual boundary beyond which we can no longer speak of anerror in position estimation, but should rather argue that thetracker has missed the Object and is Tracking something is illustrated in Fig.

10 2(a). For Object area trackers ( 1: Mapping tracker hypotheses to objects. In theeasiest case, matching the closest Object -hypothesis pairsfor each time frametis sufficienttrackers that also estimate the size of objects or the area oc-cupied by them), distance could be expressed in terms ofthe overlap between Object and hypothesis, as in [2],and the thresholdTcould be set to zero overlap. For ob-ject centroid trackers, one could simply use the Euclidiandistance, in 2D image coordinates or in real 3D world co-ordinates, between Object centers and hypotheses, and thethreshold could be, , the average width of a person inpixels or cm. In the following, we refer to correspondencesasvalidifdisti,j< Consistent Tracking over TimeSecond, to measure the tracker s ability to label objectsconsistently, one has to detect when conflicting correspon-dences have been made for an Object over time.