Multiple Object Tracking Performance Metrics and ...

1 Multiple Object Tracking Performance Metricsand Evaluation in a Smart Room EnvironmentKeni Bernardin, Alexander Elbs, Rainer StiefelhagenInstitut f ur Theoretische InformatikInteractive Systems LabUniversit at Karlsruhe, 76131 Karlsruhe, Tracking of Multiple persons in real worldenvironments is an active research field and several ap-proaches have been proposed, based on a variety of featuresand algorithms. Recently, there has been a growing interestin organizing systematic evaluations to compare the varioustechniques. Unfortunately, the lack of common Metrics formeasuring the Performance of Multiple Object trackers stillmakes it hard to compare their this work, we introduce two intuitive and general metricsto allow for objective comparison of tracker characteris-tics, focusing on their precision in estimating Object loca-tions, their accuracy in recognizing Object configurationsand their ability to consistently label objects over also present a novel system for Tracking Multiple users ina smart room environment using several cameras.

2 Based oncolor histogram Tracking of person regions and automaticinitialization using special Object system is used to demonstrate the expressiveness of theproposed Metrics through a sample Performance evaluationusing real test video sequences of people interacting in thesmart Introduction and Related WorkThe Tracking of Multiple persons in camera images is a veryactive research field with applications in many range from video surveillance, over automatic index-ing, to intelligent interactive environments. Especially inthe last case, a robust person Tracking module can serve asa poweful building block to support other techniques, suchas gesture recognizers, face identifiers, head pose estima-tors [10], scene analysis tools, etc.

3 In the last few years,more and more approaches have been presented to tacklethe problems posed by unconstrained, natural environmentsand bring person trackers out of the laboratory environmentand into real world recent times, there has also been a growing interestin performing systematic evaluations of such Tracking toolswith common databases and Metrics . Examples are theCHIL project, funded by the EU [17], the VACE projectin the [18], but also a growing number of workshops(PETS [19], EEMCV [20], etc). However, there is still nogeneral agreement on a principled evaluation procedure us-ing a common set of objective and intuitive Metrics for mea-suring the Performance of Multiple Object trackers.

4 Becauseof this lack of Metrics , some researchers present their track-ing systems without any quantitative evaluation of their per-formance ( [1, 7, 15]). On the other hand, a multitude ofisolated measures were defined in individual contributionsto validate trackers using various features and algorithms(see [2, 3, 5, 11, 13]), but no common agreement on abest set of measures remedy this, this paper proposes a thorough proce-dure to detect all types of errors produced by a Multiple ob-ject Tracking system and introduces two novel Metrics , theMultiple Object Tracking Precision (M OT P), and the Mul-tiple Object Tracking Accuracy(M OT A)

5 , that intuitivelyexpress a tracker s characteristics and could be used in gen-eral Performance the work that most closely relates to ours is thatof Smith et al. in [4]. The authors also attempt to definean objective procedure and measures for Multiple objecttracker Performance . However, key differences to our con-tribution exist: In [4], the autors introduce a large number ofmetrics: 5 for measuring Object configuration errors, and 4for measuring inconsistencies in Object labeling over of the measures are defined in a dual way for trackersand for objects ( T/M O,F IT/F IO,T P/OP).

6 Thiscould make it difficult to gain a clear and direct understand-ing of the tracker s overall Performance . Moreover, undercertain conditions, some of these measures can behave in anon-intuitive fashion (such as theCD, as the authors state,or theF PandF N, as we will demonstrate later). In com-parison, we introduce just 2 overall Performance measuresthat allow a clear and intuitive insight into the main trackercharacteristics: its precision in estimating Object positions,its ability to determine the number of objects and their con-figuration, and its skill at keeping consistent tracks overtime.

7 In addition, we offer an experimental validation ofthe presented theoretical framework by performing sampleevaluation runs on 2 variants of a Multiple person tracker,using real data recorded in a smart room environment. Ademonstration run on simulated data is also performed tobetter illustrate the expressiveness of the proposed system used in our evaluations is a 3D Multiple per-son tracker developed for use in our smart room. It initial-izes automatically using special person detectors, performscolor histogram Tracking of body parts on several cameraviews and intelligently fuses the 2D information to producea consistent set of 3D hypotheses.

8 It is used as a proof ofconcept for the introducedM OT PandM OT Ametrics,which are used to measure its accuracy on datasets of vary-ing degrees of remainder of the paper is organized as follows:Section 2 presents the new Metrics , theM OT Pand theM OT Aand a detailed procedure for their 3 briefly introduces the developed multipersontracker which will be used in the evaluations. In Section 4,the sample Performance measurements are shown and theusefulness of the Metrics is discussed. Finally, Section 5gives a summary and a Performance Metrics for MultipleObject TrackingTo help better understand the proposed evaluation Metrics ,we first explain what qualities we expect from an ideal mul-tiple Object tracker.

9 It should at all points in time find thecorrect number of objects present and estimate the positionof each Object as precisely as possible (Note that propertiessuch as the size, contour, orientation or speed of objects arenot considered here). It should also keep consistent trackof each Object over time: Each Object should be assigneda unique track ID which stays constant throughout the se-quence (even after temporary occlusion, etc). This leadsto the following design criteria for Performance evaluationmetrics: They should allow to judge the tracker s precision indetermining exact Object locations.

10 They should reflect its ability to consistently track ob-ject configurations through time, to correctly traceobject trajectories, producing exactly one trajectoryper , we expect useful Metrics to have as few free parameters, adjustable thresholds,etc, as possible to help make evaluations straightfor-ward and keep results comparable. to be clear, easily understandable and behave accord-ing to human intuition, especially in the occurence ofmultiple errors of different types or of uneven reparti-tion of errors throughout the sequence. to be general enough to allow comparison of mosttypes of trackers (2D, 3D trackers, Object centroidtrackers or Object area trackers, etc).