Indoor Segmentation and Support Inference from RGBD …

1 Indoor Segmentation and Support Inferencefrom RGBD ImagesNathan Silberman1, Derek Hoiem2, Pushmeet Kohli3, Rob Fergus11 Courant Institute, New York University2 Department of Computer Science, University of Illinois at Urbana-Champaign3 Microsoft Research, present an approach to interpret the major surfaces, ob-jects, and Support relations of an Indoor scene from an RGBD existing work ignores physical interactions or is applied only totidy rooms and hallways. Our goal is to parse typical, often messy, in-door scenes into floor, walls, supporting surfaces, and object regions, andto recover Support relationships. One of our main interests is to betterunderstand how 3D cues can best inform a structured 3D interpreta-tion. We also contribute a novel integer programming formulation toinfer physical Support relations. We offer a new dataset of 1449 RGBD images, capturing 464 diverse Indoor scenes, with detailed experiments demonstrate our ability to infer Support relations incomplex scenes and verify that our 3D scene cues and inferred supportlead to better object IntroductionTraditional approaches to scene understanding aim to provide labels for eachobject in the image.

2 However, this is an impoverished description since labelstell us little about the physical relationships between objects, possible actionsthat can be performed, or the geometric structure of the robotics and scene understanding applications require a physical parseof the scene into objects, surfaces, and their relations. A person walking into aroom, for example, might want to find his coffee cup and favorite book, grabthem, find a place to sit down, walk over, and sit down. These tasks requireparsing the scene into different objects and surfaces the coffee cup must bedistinguished from surrounding objects and the supporting surface for tasks also require understanding the interactions of scene elements: if thecoffee cup is supported by the book, then the cup must be lifted this paper, our goal is to provide such a physical scene parse: to segmentvisible regions into surfaces and objects and to infer their Support relations. Inparticular, we are interested in Indoor scenes that reflect typical living include the well-known difficulty of object Segmentation , prevalenceof small objects, and heavy occlusion, which are all compounded by the messand disorder that are common in lived-in rooms.

3 What makes interpretationpossible at all is the rich geometric structure: most rooms are composed of largeplanar surfaces, such as the floor, walls, and table tops, and objects can oftenbe interpreted in relation to those surfaces. We can better interpret the room byrectifying our visual data with the room s geometric submission ID 1079 Our approach, illustrated in Fig. 1, is to first infer the overall 3D structureof the scene and then jointly parse the image into separate objects and estimatetheir Support relations. Some tasks, such as estimating the floor orientation orfinding large planar surfaces are much easier with depth information, which iseasy to acquire indoors. But other tasks, such as segmenting and classifyingobjects require appearance based cues. Thus, we use depth cues to sidestepthe common geometric challenges that bog down single-view image-based ap-proaches, enabling a more detailed and accurate geometric structure. We arethen able to focus on properly leveraging this structure to jointly segment theobjects and infer Support relations, using both image and depth cues.

4 One of ourinnovations is to classify objects intostructural classesthat reflect their physicalrole in the scene: ground ; permanent structures such as walls, ceilings, andcolumns; large furniture such as tables, dressers, and counters; and props which are easily movable objects. We show that these structural classes aid bothsegmentation and Support reason about Support , we introduce a principled approach that integratesphysical constraints ( is the object close to its putative supporting object?)and statistical priors on Support relationships ( mugs are often supportedby tables, but rarely by walls). Our method is designed for real-world scenesthat contain tens or hundred of objects with heavy occlusion and clutter. Inthis setting, interfaces between objects are often not visible and thus must beinferred. Even without occlusion, limited image resolution can make supportambiguous, necessitating global reasoning between image regions.

5 Real-worldimages also contain significant variation in focal length. While wide-angle shotscontain many objects, narrow-angle views can also be challenging as importantstructural elements of the scene, such as the floor, are not observed. Our schemeis able to handle these situations by inferring the location of invisible elementsand how they interact with the visible components of the Related WorkOur overall approach of incorporating geometric priors to improve scene inter-pretation is most related to a set of image-based single-view methods ( [1 7]).Our use of structural classes , such as furniture and prop , to improve seg-mentation and Support Inference relates to the use of geometric classes [1]to segment objects [8] or volumetric scene parses [3, 5 7]. Our goal of inferringsupport relations is most closely related to Gupta et al. [6], who apply heuristicsinspired by physical reasoning to infer volumetric shapes, occlusion, and supportin outdoor scenes.

6 Our 3D cues provide a much stronger basis for Inference ofsupport, and our dataset enables us to train and evaluate Support predictorsthat can cope with scene clutter and invisible supporting regions. Russell andTorralba [9] show how a dataset of user-annotated scenes can be used to infer3D structure and Support ; our approach, in contrast, is fully approach to estimate geometric structure from depth cues is most closelyrelated to Zhang et al. [10]. After estimating depth from a camera on a mov-ing vehicle, Zhang et al. use RANSAC to fit a ground plane and represent 3 Dscene points relative to the ground and direction of the moving vehicle. We useECCV-12 submission ID 10793 Input RGBS urface NormalsAligned NormalsSegmentationInput DepthInpainted Depth3D PlanesSupport RelationsSupport Relationships 1. Major Surfaces 2. Surface Normals 3. Align Point Cloud Image Depth Map Segmentation RGB Image Point Cloud Regions Feature Extraction Features Support Classification {Xi,Ni} {Rj} {Fj} Structure Labels Fig.

7 1. Overview of algorithm flows from left to right. Given aninput image with raw and inpainted depth maps, we compute surface normals andalign them to the room by finding three dominant orthogonal directions. We thenfit planes to the points using RANSAC and segment them based on depth and colorgradients. Given the 3D scene structure and initial estimates of physical Support , wethen create a hierarchical Segmentation and infer the Support structure. In the surfacenormal images, the absolute value of the three normal directions is stored in the R,G, and B channels. The 3D planes are indicated by separate colors. Segmentation isindicated by red boundaries. Arrows point from the supported object to the surfacethat supports on 3D points to initialize plane fitting but also infer a segmentationand improved plane parameters using a graph cut Segmentation that accountsfor 3D position, 3D normal, and intensity gradients. Their application is pixellabeling, but ours is parsing into regions and Support relations.

8 Others, such asSilberman et al. [11] and Karayev et al. [12] use RGBD images from the Kinectfor object recognition, but do not consider tasks beyond category summarize, the most original of ourcontributionsis the Inference ofsupport relations in complex Indoor scenes. We incorporate geometric structureinferred from depth, object properties encoded in our structural classes, anddata- driven scene priors, and our approach is robust to clutter, stacked objects,and invisible supporting surfaces. We also contribute ideas for interpreting geo-metric structure from a depth image, such as graph cut Segmentation of planarsurfaces and ways to use the structure to improve Segmentation . Finally, we offera new large dataset with registered RGBD images, detailed object labels, andannotated physical Dataset for Indoor Scene UnderstandingSeveral Kinect scene datasets have recently been introduced. However, the NYUindoor scene dataset [11] has limited diversity (only 67 scenes); in the Berkeley4 ECCV-12 submission ID 1079 Scenes dataset [12] only a few objects per scene are labeled; and others suchas [13, 14] are designed for robotics applications.

9 We therefore introduce a newKinect dataset1, significantly larger and more diverse than existing dataset consists of 1449 RGBD images2, gathered from a wide rangeof commercial and residential buildings in three different US cities, comprising464 different Indoor scenes across 26 scene dense per-pixel labeling wasobtained for each image using Amazon Mechanical Turk. If a scene containedmultiple instances of an object class, each instance received a unique instancelabel, two different cups in the same image would be labeled: cup 1 andcup 2, to uniquely identify them. The dataset contains 35,064 distinct objects,spanning 894 different classes. For each of the 1449 images, Support annotationswere manually added. Each image s Support annotations consists of a set of 3-tuples: [Ri,Rj,type] whereRiis the region ID of the supported object,Rjisthe region ID of the supporting object andtypeindicates whether the Support isfrom below ( cup on a table) or from behind ( picture on a wall).

10 Examplesof the dataset are found in Fig 7 (object category labels not shown).3 Modeling the Structure of Indoor ScenesIndoor scenes are usually arranged with respect to the orthogonal orientationsof the floor and walls and the major planar surfaces such as supporting surfaces,floor, walls, and blocky furnishings. We treat initial Inference of scene surfaces asan alignment and Segmentation problem. We first compute surface normals fromthe depth image. Then, based on surface normals and straight lines, we find threedominant and orthogonal scene directions and rotate the 3D coordinates to beaxis aligned with the principal directions. Finally, we propose 3D planes usingRANSAC on the 3D points and segment the visible regions into one of theseplanes or background using graph cuts based on surface normals, 3D points, andRGB gradients. Several examples are shown in Fig. 2. We now describe eachstage of this procedure in more Aligning to Room CoordinatesWe are provided with registered RGB and depth images, with in-painted depthpixels [15].