Indoor Segmentation and Support Inference from RGBD …

1 Indoor Segmentation and Support Inferencefrom RGBD ImagesNathan Silberman1, Derek Hoiem2, Pushmeet Kohli3, Rob Fergus11 Courant Institute, New York University2 Department of Computer Science, University of Illinois at Urbana-Champaign3 Microsoft Research, present an approach to interpret the major surfaces, ob-jects, and Support relations of an Indoor scene from an RGBD existing work ignores physical interactions or is applied only totidy rooms and hallways. Our goal is to parse typical, often messy, in-door scenes into floor, walls, supporting surfaces, and object regions, andto recover Support relationships. One of our main interests is to betterunderstand how 3D cues can best inform a structured 3D interpreta-tion.

2 We also contribute a novel integer programming formulation toinfer physical Support relations. We offer a new dataset of 1449 RGBD images, capturing 464 diverse Indoor scenes, with detailed experiments demonstrate our ability to infer Support relations incomplex scenes and verify that our 3D scene cues and inferred supportlead to better object IntroductionTraditional approaches to scene understanding aim to provide labels for eachobject in the image. However, this is an impoverished description since labelstell us little about the physical relationships between objects, possible actionsthat can be performed, or the geometric structure of the robotics and scene understanding applications require a physical parseof the scene into objects, surfaces, and their relations.

3 A person walking into aroom, for example, might want to find his coffee cup and favorite book, grabthem, find a place to sit down, walk over, and sit down. These tasks requireparsing the scene into different objects and surfaces the coffee cup must bedistinguished from surrounding objects and the supporting surface for tasks also require understanding the interactions of scene elements: if thecoffee cup is supported by the book, then the cup must be lifted this paper, our goal is to provide such a physical scene parse: to segmentvisible regions into surfaces and objects and to infer their Support relations. Inparticular, we are interested in Indoor scenes that reflect typical living include the well-known difficulty of object Segmentation , prevalenceof small objects, and heavy occlusion, which are all compounded by the messand disorder that are common in lived-in rooms.

4 What makes interpretationpossible at all is the rich geometric structure: most rooms are composed of largeplanar surfaces, such as the floor, walls, and table tops, and objects can oftenbe interpreted in relation to those surfaces. We can better interpret the room byrectifying our visual data with the room s geometric submission ID 1079 Our approach, illustrated in Fig. 1, is to first infer the overall 3D structureof the scene and then jointly parse the image into separate objects and estimatetheir Support relations. Some tasks, such as estimating the floor orientation orfinding large planar surfaces are much easier with depth information, which iseasy to acquire indoors.

5 But other tasks, such as segmenting and classifyingobjects require appearance based cues. Thus, we use depth cues to sidestepthe common geometric challenges that bog down single-view image-based ap-proaches, enabling a more detailed and accurate geometric structure. We arethen able to focus on properly leveraging this structure to jointly segment theobjects and infer Support relations, using both image and depth cues. One of ourinnovations is to classify objects intostructural classesthat reflect their physicalrole in the scene: ground ; permanent structures such as walls, ceilings, andcolumns; large furniture such as tables, dressers, and counters; and props which are easily movable objects.

6 We show that these structural classes aid bothsegmentation and Support reason about Support , we introduce a principled approach that integratesphysical constraints ( is the object close to its putative supporting object?)and statistical priors on Support relationships ( mugs are often supportedby tables, but rarely by walls). Our method is designed for real-world scenesthat contain tens or hundred of objects with heavy occlusion and clutter. Inthis setting, interfaces between objects are often not visible and thus must beinferred. Even without occlusion, limited image resolution can make supportambiguous, necessitating global reasoning between image regions.

7 Real-worldimages also contain significant variation in focal length. While wide-angle shotscontain many objects, narrow-angle views can also be challenging as importantstructural elements of the scene, such as the floor, are not observed. Our schemeis able to handle these situations by inferring the location of invisible elementsand how they interact with the visible components of the Related WorkOur overall approach of incorporating geometric priors to improve scene inter-pretation is most related to a set of image-based single-view methods ( [1 7]).Our use of structural classes , such as furniture and prop , to improve seg-mentation and Support Inference relates to the use of geometric classes [1]to segment objects [8] or volumetric scene parses [3, 5 7].

8 Our goal of inferringsupport relations is most closely related to Gupta et al. [6], who apply heuristicsinspired by physical reasoning to infer volumetric shapes, occlusion, and supportin outdoor scenes. Our 3D cues provide a much stronger basis for Inference ofsupport, and our dataset enables us to train and evaluate Support predictorsthat can cope with scene clutter and invisible supporting regions. Russell andTorralba [9] show how a dataset of user-annotated scenes can be used to infer3D structure and Support ; our approach, in contrast, is fully approach to estimate geometric structure from depth cues is most closelyrelated to Zhang et al. [10].

9 After estimating depth from a camera on a mov-ing vehicle, Zhang et al. use RANSAC to fit a ground plane and represent 3 Dscene points relative to the ground and direction of the moving vehicle. We useECCV-12 submission ID 10793 Input RGBS urface NormalsAligned NormalsSegmentationInput DepthInpainted Depth3D PlanesSupport RelationsSupport Relationships 1. Major Surfaces 2. Surface Normals 3. Align Point Cloud Image Depth Map Segmentation RGB Image Point Cloud Regions Feature Extraction Features Support Classification {Xi,Ni} {Rj} {Fj} Structure Labels Fig. 1. Overview of algorithm flows from left to right. Given aninput image with raw and inpainted depth maps, we compute surface normals andalign them to the room by finding three dominant orthogonal directions.

10 We thenfit planes to the points using RANSAC and segment them based on depth and colorgradients. Given the 3D scene structure and initial estimates of physical Support , wethen create a hierarchical Segmentation and infer the Support structure. In the surfacenormal images, the absolute value of the three normal directions is stored in the R,G, and B channels. The 3D planes are indicated by separate colors. Segmentation isindicated by red boundaries. Arrows point from the supported object to the surfacethat supports on 3D points to initialize plane fitting but also infer a segmentationand improved plane parameters using a graph cut Segmentation that accountsfor 3D position, 3D normal, and intensity gradients.