KinectFusion: Real-time 3D Reconstruction and Interaction ...

1 kinectfusion : Real-time 3D Reconstruction and InteractionUsing a Moving Depth Camera*Shahram Izadi1, David Kim1,3, Otmar Hilliges1, David Molyneaux1,4, Richard Newcombe2,Pushmeet Kohli1, Jamie Shotton1, Steve Hodges1, Dustin Freeman1,5,Andrew Davison2, Andrew Fitzgibbon11 Microsoft Research Cambridge, UK2 Imperial College London, UK3 Newcastle University, UK4 Lancaster University, UK5 University of Toronto, CanadaFigure 1: kinectfusion enables Real-time detailed 3D reconstructions of indoor scenes using only the depth data from astandard Kinect camera. A) user points Kinect at coffee table scene.

2 B) Phong shaded reconstructed 3D model (the wireframefrustum shows current tracked 3D pose of Kinect). C) 3D model texture mapped using Kinect RGB data with Real-time particlessimulated on the 3D model as Reconstruction occurs. D) Multi-touch interactions performed on any reconstructed surface. E) Real-time segmentation and 3D tracking of a physical enables a user holding and moving a standardKinect camera to rapidly create detailed 3D reconstructionsof an indoor scene. Only the depth data from Kinect is usedtotrackthe 3D pose of the sensor andreconstruct, geomet-rically precise, 3D models of the physical scene capabilities of kinectfusion , as well as the novel GPU-based pipeline are described in full.

3 We show uses of the coresystem for low-cost handheld scanning, and geometry-awareaugmented reality and physics-based interactions . Novel ex-tensions to the core GPU pipeline demonstrate object seg-mentation and user Interaction directly in front of the sensor,without degrading camera tracking or Reconstruction . Theseextensions are used to enable Real-time multi-touch interac-tionsanywhere, allowing any planar or non-planar recon-structed physical surface to be appropriated for [Information Interfaces and Pre-sentation]: User Interfaces. [Image Processing and Com-puter Vision]: Reconstruction .

4 [Computer Graphics]:Three-Dimensional Graphics and terms:Algorithms, Design, Human :3D, GPU, Surface Reconstruction , Tracking,Depth Cameras, AR, Physics, Geometry-Aware interactions * Research conducted at Microsoft Research Cambridge, UKPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a 11,October 16-19, 2011, Santa Barbara, CA, 2011 ACM 978-1-4503-0716-1/11 $ depth cameras are not conceptually new, Kinect hasmade such sensors accessible to all.

5 The quality of the depthsensing, given the low-cost and Real-time nature of the de-vice, is compelling, and has made the sensor instantly popu-lar with researchers and enthusiasts Kinect camera uses a structured light technique [8] togenerate real -timedepth mapscontaining discrete range mea-surements of the physical scene. This data can be repro-jected as a set of discrete 3D points (orpoint cloud). Eventhough the Kinect depth data is compelling, particularly com-pared to other commercially available depth cameras, it isstill inherently noisy (see Figures2B and3left).

6 Depth mea-surements often fluctuate and depth maps contain numerous holes where no readings were generate 3D models for use in applications such as gam-ing, physics, or CAD, higher-levelsurfacegeometry needsto be inferred from this noisy point-based data. One simpleapproach makes strong assumptions about the connectivityof neighboring points within the Kinect depth map to gen-erate ameshrepresentation. This, however, leads to noisyand low-quality meshes as shown in Figure2C. As impor-tantly, this approach creates an incomplete mesh, from onlya single, fixed viewpoint.

7 To create a complete (or even wa-tertight) 3D model, different viewpoints of the physical scenemust be captured andfusedinto a single paper presents a novel interactive Reconstruction sys-tem called kinectfusion (see Figure1). The system takeslive depth data from a moving Kinect camera and, in Real-time , creates a single high-quality, geometrically accurate,3D model. A user holding a standard Kinect camera canmove within any indoor space, and reconstruct a 3D modelof the physical scene within seconds. The system contin-Figure 2: RGB image of scene (A). Extracted normals (B) and surface Reconstruction (C) from a single bilateral filtered Kinectdepth map.

8 3D model generated from kinectfusion showing surface normals (D) and rendered with Phong shading (E).uously tracks the 6 degrees-of-freedom (DOF) pose of thecamera and fuses new viewpoints of the scene into a globalsurface-based representation. A novel GPU pipeline allowsfor accurate camera tracking and surface Reconstruction at in-teractive Real-time rates. This paper details the capabilities ofour novel system, as well as the implementation of the GPUpipeline in demonstrate core uses of kinectfusion as a low-costhandheld scanner, and present novel interactive methods forsegmentingphysical objects of interest from the reconstructedscene.

9 We show how a Real-time 3D model can be leveragedforgeometry-awareaugmented reality (AR) and physics-based interactions , where virtual worlds more realisticallymerge and interact with the such systems into an Interaction context, where usersneed to dynamically interact in front of the sensor, reveals afundamental challenge no longer can we assume a staticscene for camera tracking or Reconstruction . We illustratefailure cases caused by a user moving in front of the describe new methods to overcome these limitations, al-lowing camera tracking and Reconstruction of a static back-ground scene, while simultaneously segmenting, reconstruct-ing and tracking foreground objects, including the user.

10 Weuse this approach to demonstrate Real-time multi-touch inter-actionsanywhere, allowing a user to appropriate any physicalsurface, be it planar or non-planar, for WORKR econstructing geometry using active sensors [16], passivecameras [11,18], online images [7], or from unordered 3 Dpoints [14,29] are well-studied areas of research in com-puter graphics and vision. There is also extensive literaturewithin the AR and robotics community on Simultaneous Lo-calization and Mapping (SLAM), aimed at tracking a user orrobot while creating a map of the surrounding physical envi-ronment (see [25]).