KinectFusion: Real-time 3D Reconstruction and Interaction ...

1 kinectfusion : Real-time 3D Reconstruction and InteractionUsing a Moving Depth Camera*Shahram Izadi1, David Kim1,3, Otmar Hilliges1, David Molyneaux1,4, Richard Newcombe2,Pushmeet Kohli1, Jamie Shotton1, Steve Hodges1, Dustin Freeman1,5,Andrew Davison2, Andrew Fitzgibbon11 Microsoft Research Cambridge, UK2 Imperial College London, UK3 Newcastle University, UK4 Lancaster University, UK5 University of Toronto, CanadaFigure 1: kinectfusion enables Real-time detailed 3D reconstructions of indoor scenes using only the depth data from astandard Kinect camera. A) user points Kinect at coffee table scene. B) Phong shaded reconstructed 3D model (the wireframefrustum shows current tracked 3D pose of Kinect). C) 3D model texture mapped using Kinect RGB data with Real-time particlessimulated on the 3D model as Reconstruction occurs. D) Multi-touch interactions performed on any reconstructed surface. E) Real-time segmentation and 3D tracking of a physical enables a user holding and moving a standardKinect camera to rapidly create detailed 3D reconstructionsof an indoor scene.

2 Only the depth data from Kinect is usedtotrackthe 3D pose of the sensor andreconstruct, geomet-rically precise, 3D models of the physical scene capabilities of kinectfusion , as well as the novel GPU-based pipeline are described in full. We show uses of the coresystem for low-cost handheld scanning, and geometry-awareaugmented reality and physics-based interactions . Novel ex-tensions to the core GPU pipeline demonstrate object seg-mentation and user Interaction directly in front of the sensor,without degrading camera tracking or Reconstruction . Theseextensions are used to enable Real-time multi-touch interac-tionsanywhere, allowing any planar or non-planar recon-structed physical surface to be appropriated for [Information Interfaces and Pre-sentation]: User Interfaces. [Image Processing and Com-puter Vision]: Reconstruction . [Computer Graphics]:Three-Dimensional Graphics and terms:Algorithms, Design, Human :3D, GPU, Surface Reconstruction , Tracking,Depth Cameras, AR, Physics, Geometry-Aware interactions * Research conducted at Microsoft Research Cambridge, UKPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page.

3 To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a 11,October 16-19, 2011, Santa Barbara, CA, 2011 ACM 978-1-4503-0716-1/11 $ depth cameras are not conceptually new, Kinect hasmade such sensors accessible to all. The quality of the depthsensing, given the low-cost and Real-time nature of the de-vice, is compelling, and has made the sensor instantly popu-lar with researchers and enthusiasts Kinect camera uses a structured light technique [8] togenerate real -timedepth mapscontaining discrete range mea-surements of the physical scene. This data can be repro-jected as a set of discrete 3D points (orpoint cloud). Eventhough the Kinect depth data is compelling, particularly com-pared to other commercially available depth cameras, it isstill inherently noisy (see Figures2B and3left). Depth mea-surements often fluctuate and depth maps contain numerous holes where no readings were generate 3D models for use in applications such as gam-ing, physics, or CAD, higher-levelsurfacegeometry needsto be inferred from this noisy point-based data.

4 One simpleapproach makes strong assumptions about the connectivityof neighboring points within the Kinect depth map to gen-erate ameshrepresentation. This, however, leads to noisyand low-quality meshes as shown in Figure2C. As impor-tantly, this approach creates an incomplete mesh, from onlya single, fixed viewpoint. To create a complete (or even wa-tertight) 3D model, different viewpoints of the physical scenemust be captured andfusedinto a single paper presents a novel interactive Reconstruction sys-tem called kinectfusion (see Figure1). The system takeslive depth data from a moving Kinect camera and, in Real-time , creates a single high-quality, geometrically accurate,3D model. A user holding a standard Kinect camera canmove within any indoor space, and reconstruct a 3D modelof the physical scene within seconds. The system contin-Figure 2: RGB image of scene (A). Extracted normals (B) and surface Reconstruction (C) from a single bilateral filtered Kinectdepth map.

5 3D model generated from kinectfusion showing surface normals (D) and rendered with Phong shading (E).uously tracks the 6 degrees-of-freedom (DOF) pose of thecamera and fuses new viewpoints of the scene into a globalsurface-based representation. A novel GPU pipeline allowsfor accurate camera tracking and surface Reconstruction at in-teractive Real-time rates. This paper details the capabilities ofour novel system, as well as the implementation of the GPUpipeline in demonstrate core uses of kinectfusion as a low-costhandheld scanner, and present novel interactive methods forsegmentingphysical objects of interest from the reconstructedscene. We show how a Real-time 3D model can be leveragedforgeometry-awareaugmented reality (AR) and physics-based interactions , where virtual worlds more realisticallymerge and interact with the such systems into an Interaction context, where usersneed to dynamically interact in front of the sensor, reveals afundamental challenge no longer can we assume a staticscene for camera tracking or Reconstruction .

6 We illustratefailure cases caused by a user moving in front of the describe new methods to overcome these limitations, al-lowing camera tracking and Reconstruction of a static back-ground scene, while simultaneously segmenting, reconstruct-ing and tracking foreground objects, including the user. Weuse this approach to demonstrate Real-time multi-touch inter-actionsanywhere, allowing a user to appropriate any physicalsurface, be it planar or non-planar, for WORKR econstructing geometry using active sensors [16], passivecameras [11,18], online images [7], or from unordered 3 Dpoints [14,29] are well-studied areas of research in com-puter graphics and vision. There is also extensive literaturewithin the AR and robotics community on Simultaneous Lo-calization and Mapping (SLAM), aimed at tracking a user orrobot while creating a map of the surrounding physical envi-ronment (see [25]). Given this broad topic, and our desireto build a system forinteraction, this section is structuredaround specific design goals that differentiate KinectFusionfrom prior work.

7 The combination of these features makesour interactive Reconstruction system ratesOur primary goal with kinectfusion is toachieve Real-time interactive rates forbothcamera trackingand3D Reconstruction . This speed is critical for permittingdirect feedback and user Interaction . This differentiates usfrom many existing Reconstruction systems that support onlyoffline reconstructions [7], Real-time but non-interactive rates( the Kinect-based system of [12] reconstructs at 2Hz),or support Real-time camera tracking but non Real-time recon-struction or mapping phases [15,19,20].No explicit feature detectionUnlike structure from mo-tion (SfM) systems ( [15]) or RGB plus depth (RGBD)techniques ( [12,13]), which need to robustly and con-tinuously detect sparse scene features, our approach to cam-era tracking avoids an explicit detection step, and directlyworks on the full depth maps acquired from the Kinect sen-sor.

8 Our system also avoids the reliance on RGB (used inrecent Kinect RGBD systems [12]) allowing use in in-door spaces with variable lighting Reconstruction of geometryA core goal ofour work is to capture detailed (ordense) 3D models ofthe real scene. Many SLAM systems ( [15]) focuson Real-time tracking, usingsparsemaps for localizationrather than Reconstruction . Others have used simple point-based representations (such as surfels [12] or aligned point-clouds [13]) for Reconstruction . kinectfusion goes beyondthese point-based representations by reconstructingsurfaces,which more accurately approximate real -world Interaction assumedWe explore tracking andreconstruction in the context ofuser Interaction . Given thisrequirement, it is critical that the representation we use candeal with dynamically changing scenes, where users directlyinteract in front of the camera. While there has been workon using mesh-based representations for live reconstructionfrom passive RGB [18,19,20] or active time -of-Flight (ToF)cameras [4,28], these do not readily deal with changing, dy-namic aim to allow users to explore andreconstruct arbitrary indoor spaces.

9 This suggests a level ofmobility, and contrasts with systems that use fixed or largesensors ( [16,23]) or are fully embedded in the envi-ronment ( [26]). We also aim to perform camera track-ing without the need for prior augmentation of the space,whether this is the use of infrastructure-heavy tracking sys-tems ( [2]) or fiducial markers ( [27]).Room scaleOne final requirement is to support wholeroom reconstructions and Interaction . This differentiatesKinectFusion from prior dense Reconstruction systems, whichhave either focused on smaller desktop scenes [19,20] orscanning of small physical objects [4,28].The remainder of this paper is structured into two parts: Thefirst provides a high-level description of the capabilities ofKinectFusion. The second describes the technical aspects ofthe system, focusing on our novel GPU 3: Left: Raw Kinect data (shown as surface nor-mals). Right: Reconstruction shows hole filling and high-quality details such as keys on keyboard, phone numberpad, wires, and even a DELL logo on the side of a PC (anengraving less than 1mm deep).

10 Figure 4: A) User rotating object in front of fixed ) 360 3D Reconstruction . C) 3D model imported intoSolidWorks. D) 3D printout from system allows a user to pickup a standard Kinect cameraand move rapidly within a room to reconstruct a high-quality,geometrically precise 3D model of the scene. To achieve this,our system continually tracks the 6 DOF pose of the cameraand fuses live depth data from the camera into a single global3D model in Real-time . As the user explores the space, newviews of the physical scene are revealed and these are fusedinto the same model. The Reconstruction therefore grows indetail as new depth measurements are added. Holes are filled,and the model becomes more complete and refined over time (see Figure3).Even small motions, caused for example by camera shake,result in new viewpoints of the scene and hence refinementsto the model. This creates an effect similar to image super-resolution [6] adding greater detail than appears visible inthe raw signal (see Figure3).