A Database and Evaluation Methodology for Optical Flow

1 Int J Comput Vis (2011) 92: 1 31 DOI Database and Evaluation Methodology for Optical FlowSimon Baker Daniel Scharstein Lewis Stefan Roth Michael J. Black Richard SzeliskiReceived: 18 December 2009 / Accepted: 20 September 2010 / Published online: 30 November 2010 The Author(s) 2010. This article is published with open access at quantitative Evaluation of Optical flow algo-rithms by Barron et al. (1994) led to significant advancesin performance. The challenges for Optical flow algorithmstoday go beyond the datasets and Evaluation methods pro-posed in that paper. Instead, they center on problems as-sociated with complex natural scenes, including nonrigidmotion, real sensor noise, and motion discontinuities.

2 Wepropose a new set of benchmarks and Evaluation methodsfor the next generation of Optical flow algorithms. To thatend, we contribute four types of data to test different as-pects of Optical flow algorithms: (1) sequences with non-rigid motion where the ground-truth flow is determined byA preliminary version of this paper appeared in the IEEE InternationalConference on Computer Vision (Baker et ).S. Baker R. SzeliskiMicrosoft Research, Redmond, WA, USAS. Scharstein ( )Middlebury College, Middlebury, VT, LewisWeta Digital, Wellington, New RothTU Darmstadt, Darmstadt, BlackBrown University, Providence, RI, hidden fluorescent texture, (2) realistic syntheticsequences, (3) high frame-rate video used to study inter-polation error, and (4) modified stereo sequences of staticscenes.

3 In addition to the average angular error used by Bar-ron et al., we compute the absolute flow endpoint error, mea-sures for frame interpolation error, improved statistics, andresults at motion discontinuities and in textureless October 2007, we published the performance of severalwell-known methods on a preliminary version of our datato establish the current state of the art. We also made thedata freely available on the web Subsequently a number of researchers have up-loaded their results to our website and published papers us-ing the data. A significant improvement in performance hasalready been achieved. In this paper we analyze the resultsobtained to date and draw a large number of conclusionsfrom flow Survey Algorithms Database Benchmarks Evaluation Metrics1 IntroductionAs a subfield of computer vision matures, datasets forquantitatively evaluating algorithms are essential to ensurecontinued progress.

4 Many areas of computer vision, suchas stereo (Scharstein and Szeliski2002), face recognition(Philips et ; Sim et ; Gross et ;Georghiades et ), and object recognition (Fei-Feiet ; Everingham et ), have challengingdatasets to track the progress made by leading algorithmsand to stimulate new ideas. Optical flow was actually oneof the first areas to have such a benchmark, introduced byBarron et al. (1994). The field benefited greatly from this2 Int J Comput Vis (2011) 92: 1 31study, which led to rapid and measurable progress. To con-tinue the rapid progress, new and more challenging datasetsare needed to push the limits of current technology, revealwhere current algorithms fail, and evaluate the next gener-ation of Optical flow algorithms.

5 Such an Evaluation datasetfor Optical flow should ideally consist of complex real sceneswith all the artifacts of real sensors (noise, motion blur, etc.).It should also contain substantial motion discontinuities andnonrigid motion. Of course, the image data should be pairedwith dense, subpixel-accurate, ground-truth flow presence of nonrigid or independent motion makescollecting a ground-truth dataset for Optical flow far harderthan for stereo, say, where structured light (Scharstein andSzeliski2002) or range scanning (Seitz et ) canbe used to obtain ground truth. Our solution is to collectfour different datasets, each satisfying a different subset ofthe desirable properties above.

6 The combination of thesedatasets provides a basis for a thorough Evaluation of currentoptical flow algorithms. Moreover, the relative performanceof algorithms on the different datatypes may stimulate fur-ther research. In particular, we collected the following fourtypes of data: Real Imagery of Nonrigidly Moving Scenes:Denseground-truth flow is obtained using hidden fluorescenttexture painted on the scene. We slowly move the scene,at each point capturing separate test images (in visiblelight) and ground-truth images with trackable texture (inUV light). Note that a related technique is being usedcommercially for motion capture (Mova LLC2004) andTappen et al.

7 (2006) recently used certain wavelengthsto hide ground truth in intrinsic images. Another form ofhidden markers was also used in Ramnath et al. (2008)toprovide a sparse ground-truth alignment (or flow) of faceimages. Finally, Liu et al. recently proposed a method toobtain ground-truth using human annotation (Liu et ). Realistic Synthetic Imagery: We address the limitations ofsimple synthetic sequences such asYosemite(Barron et ) by rendering more complex scenes with larger mo-tion ranges, more realistic texture, independent motion,and with more complex occlusions. Imagery for Frame Interpolation: Intermediate frames arewithheld and used as ground truth.

8 In a wide class of ap-plications such as video re-timing, novel-view generation,and motion-compensated compression, what is importantis not how well the flow matches the ground-truth motion,but how well intermediate frames can be predicted usingthe flow (Szeliski1999). Real Stereo Imagery of Rigid Scenes: Dense ground truthis captured using structured light (Scharstein and Szeliski2003). The data is then adapted to be more appropriatefor Optical flow by cropping to make the disparity rangeroughly collected enough data to be able to split our collec-tion into a training set (12 datasets) and a final evalua-tion set (12 datasets).

9 The training set includes the groundtruth and is meant to be used for debugging, parameterestimation, and possibly even learning (Sun et ;Li and Huttenlocher2008). The ground truth for the finalevaluation set is not publicly available (with the exceptionof theYosemitesequence, which is included in the test set toallow some comparison with algorithms published prior tothe release of our data).We also extend the set of performance measures and theevaluation Methodology of Barron et al. (1994) to focus at-tention on current algorithmic problems: Error Metrics: We report both average angular error (Bar-ron et ) and flow endpoint error (pixel distance)(Otte and Nagel1994).

10 For image interpolation, we com-pute the residual RMS error between the interpolated im-age and the ground-truth image. We also report a gradient-normalized RMS error (Szeliski1999). Statistics: In addition to computing averages and standarddeviations as in Barron et al. (1994), we also computerobustness measures (Scharstein and Szeliski2002) andpercentile-based accuracy measures (Seitz et ). Region Masks: Following Scharstein and Szeliski (2002),we compute the error measures and their statistics overcertain masked regions of research interest. In particular,we compute the statistics near motion discontinuities andin textureless that we require flow algorithms to estimate a denseflow field.