KinectFusion: Real-Time Dense Surface Mapping and …

1 kinectfusion : Real-Time Dense Surface Mapping and Tracking Richard A. NewcombeImperial College LondonShahram IzadiMicrosoft ResearchOtmar HilligesMicrosoft ResearchDavid MolyneauxMicrosoft ResearchLancaster UniversityDavid KimMicrosoft ResearchNewcastle UniversityAndrew J. DavisonImperial College LondonPushmeet KohliMicrosoft ResearchJamie ShottonMicrosoft ResearchSteve HodgesMicrosoft ResearchAndrew FitzgibbonMicrosoft ResearchFigure 1: Example output from our system, generated in Real-Time with a handheld Kinect depth camera and no other sensing maps (colour) and Phong-shaded renderings (greyscale) from our Dense reconstruction system are shown. On the left for comparisonis an example of the live, incomplete, and noisy data from the Kinect sensor (used as input to our system).ABSTRACTWe present a system for accurate Real-Time Mapping of complex andarbitrary indoor scenes in variable lighting conditions, using only amoving low-cost depth camera and commodity graphics fuse all of the depth data streamed from a Kinect sensor intoa single global implicit Surface model of the observed scene inreal- time .

2 The current sensor pose is simultaneously obtained bytracking the live depth frame relative to the global model using acoarse-to-fine iterative closest point (ICP) algorithm, which usesall of the observed depth data available. We demonstrate the advan-tages of tracking against the growing full Surface model comparedwith frame-to-frame tracking, obtaining tracking and Mapping re-sults in constant time within room sized scenes with limited driftand high accuracy. We also show both qualitative and quantitativeresults relating to various aspects of our tracking and Mapping sys-tem. Modelling of natural scenes, in Real-Time with only commod-ity sensor and GPU hardware, promises an exciting step forwardin augmented reality (AR), in particular, it allows Dense surfaces tobe reconstructed in Real-Time , with a level of detail and robustnessbeyond any solution yet presented using passive computer : Real-Time , Dense reconstruction , Tracking, GPU,SLAM, Depth Cameras, Volumetric Representation, ARIndex [Computer Graphics] Picture/Image Genera-tion - Digitizing and Scanning; [Image Processing and Com-puter Vision] Scene Analysis - Tracking, Surface Fitting.

3 [Information Interfaces and Presentation]: Multimedia InformationSystems - Artificial, augmented, and virtual realities This work was performed at Microsoft INTRODUCTIONReal- time infrastructure-free tracking of a handheld camera whilstsimultaneously Mapping the physical scene in high-detail promisesnew possibilities for augmented and mixed reality computer vision, research on structure from motion (SFM)and multi - view stereo (MVS) has produced many compelling re-sults, in particular accurate camera tracking and sparse reconstruc-tions ( [10]), and increasingly reconstruction ofdensesurfaces( [24]). However, much of this work was not motivated by Real-Time on simultaneous localisation and Mapping (SLAM) hasfocused more on Real-Time markerless tracking andlivescene re-construction based on the input of a single commodity sensor amonocular RGB camera. Such monocular SLAM systems such asMonoSLAM [8] and the more accurate Parallel Tracking and Map-ping (PTAM) system [17] allow researchers to investigate flexibleinfrastructure- and marker-free AR applications.

4 But while thesesystems perform Real-Time Mapping , they were optimised for ef-ficient camera tracking, with the sparse point cloud models theyproduce enabling only rudimentary scene the past year, systems have begun to emerge that combinePTAM s handheld camera tracking capability with Dense surfaceMVS-style reconstruction modules, enabling more sophisticatedocclusion prediction and Surface interaction [19, 26]. Most recentlyin this line of research, iterative image alignment against Dense re-constructions has also been used to replace point features for cam-era tracking [20]. While this work is very promising for AR, densescene reconstruction in Real-Time remains a challenge for passivemonocular systems which assume the availability of the right typeof camera motion and suitable scene while algorithms for estimating camera pose and extract-ing geometry from images have been evolving at pace, so havethe camera technologies themselves.

5 New depth cameras based ei-ther on time -of-flight (ToF) or structured light sensing offer densemeasurements of depth in an integrated device. With the arrivalof Microsoft s Kinect, such sensing has suddenly reached wideconsumer-level accessibility. The opportunities for SLAM and ARFigure 2: A larger scale reconstruction obtained in such sensors are obvious, but algorithms to date have not fullyleveraged the fidelity and speed of sensing that such devices this paper we present a detailed method with analysis of whatwe believe is the first system which permits Real-Time , Dense volu-metric reconstruction of complex room-sized scenes using a hand-held Kinect depth sensor. Users can simply pick up and move aKinect device to generate a continuously updating, smooth, fullyfused 3D Surface reconstruction . Using only depth data, the systemcontinuously tracks the 6 degrees-of-freedom (6 DOF) pose of thesensorusing all of the live data availablefrom the Kinect sensorrather than an abstracted feature subset, and integrates depth mea-surements into a global Dense volumetric model.

6 A key noveltyis that tracking, performed at 30Hz frame-rate, is always relativeto thefully up-to-date fused Dense model, and we demonstrate theadvantages this offers. By using only depth data, our system canwork in complete darkness mitigating any issues concerning lowlight conditions, problematic for passive camera [17, 19, 26] andRGB-D based systems [14]. Examples of live reconstructions gen-erated from our system are shown throughout the paper ranging inscale including Figures 1 and use of highly parallel general purpose GPU (GPGPU) tech-niques is at the core of all of our design decisions, allowing bothtracking and Mapping to perform at the frame-rate of the Kinectsensor (and even at higher rates) and in constant time . Qualitativeand quantitative results in this paper demonstrate various aspects ofthe tracking and Mapping system believe that the resulting system will become an enabler formany AR and interaction scenarios.

7 In particular, the high quality, real time , reconstruction and tracking will enable a fuller physicallypredictive interaction between the virtual and the real scene, as wellas providing the high quality occlusion handling required for com-plex mixed reality The Kinect SensorKinect is a new and widely-available commodity sensor platformthat incorporates a structured light based depth sensor. Using anon-board ASIC a 11-bit 640x480 depth map is generated at the quality of this depth map is generally remarkable giventhe cost of the device, a number of challenges remain. In particular,the depth images contain numerous holes where no structured-light depth reading was possible. This can be due to certain ma-terials or scene structures which do not reflect infra-red (IR) light,very thin structures or surfaces at glancing incidence angles. Whenmoved fast the device will also experience motion blur (like anycamera) and this can also lead to missing system takes the Real-Time stream of noisy depth maps fromKinect and performs Real-Time Dense SLAM, producing a consistent3D scene model incrementally while simultaneously tracking thesensor s agile motion using all of the depth data in each frame.

8 Inthe following sections we review important related work on SLAM, Dense tracking, Surface representations and previous work on jointtracking and modelling with active depth Drift-Free SLAM for ARMost SLAM algorithms must be capable of producing self-consistent scene maps and performing drift-free sensor tracking ina sequential, Real-Time fashion. Early SFM algorithms capable ofdealing with a large number of images had either tracked cameramotion incrementally, accumulating drift [2], or required off-lineoptimisation [10] to close loops. The first monocular SLAM sys-tem capable of producing a globally consistent maps in real -timewith a handheld camera was based on probabilistic filtering of ajoint state consisting of camera and scene feature position estimates[8]. This system was targeted at small-scale workspaces compatiblewith some AR applications, but was in fact limited to these due thehigh computational cost of filtering a large state vector containingthe many features that would be needed to map larger areas.

9 Evenin small spaces, this issue meant that the tracking accuracy whichcould practically be achieved in Real-Time was relatively poor dueto the sparse feature maps , it was discovered to be practically advantageous to aban-don the propagation of a full probabilistic state and instead to runtwo procedures in alternation or in parallel:tracking, estimating thepose of the sensor on the assumption that the current scene modelis perfectly accurate; andmapping, improving and expanding themap using a form of global optimisation. This approach was pi-oneered by the PTAM system [17] which demonstrated qualityreal- time SLAM with a monocular camera in small s Mapping component is nothing but bundle adjustment, theclassical least-squares solution to camera and feature optimisation,but implemented judiciously over an automatically selected set ofspatially-distributed keyframes and running repeatedly as often ascomputing resources will allow.

10 PTAM s highly engineered livetracking component runs in parallel at frame-rate, and performs fea-ture matching and robustn-point pose estimation. Compared to fil-ters, this architecture means that an order of magnitude more scenefeatures could be packed into the map [25], and the result was Real-Time accuracy now comparable to results in off-line PTAM s principle of splitting tracking and Mapping can betaken much further, since it allows a flexible choice of the com-ponents used for each of those processes. PTAM, as a feature-based system, achieves excellent accuracy for camera tracking inpreviously unknown scenes, but the sparse pointmapit generates isstill not useful for much beyond providing localisation number of recent approaches have concentrated on this point,and have demonstrated live reconstruction of Dense geometry usingmulti- view stereo techniques while relying on sparse models forestimating the sensor motion.