PointNet: Deep Learning on Point Sets for 3D ...

1 PointNet: Deep Learning on Point Sets for 3D Classification and SegmentationCharles R. Qi*Hao Su*Kaichun MoLeonidas J. GuibasStanford UniversityAbstractPoint cloud is an important type of geometric datastructure. Due to its irregular format, most researcherstransform such data to regular 3D voxel grids or collectionsof images. This, however, renders data unnecessarilyvoluminous and causes issues. In this paper , we design anovel type of neural network that directly consumes pointclouds, which well respects the permutation invariance ofpoints in the input. Our network, named PointNet, pro-vides a unified architecture for applications ranging fromobject classification, part segmentation, to scene semanticparsing. Though simple, PointNet is highly efficient andeffective. Empirically, it shows strong performance onpar or even better than state of the art. Theoretically,we provide analysis towards understanding of what thenetwork has learnt and why the network is robust withrespect to input perturbation and IntroductionIn this paper we explore deep Learning architecturescapable of reasoning about 3D geometric data such aspoint clouds or meshes.

2 Typical convolutional architecturesrequire highly regular input data formats, like those ofimage grids or 3D voxels, in order to perform weightsharing and other kernel optimizations. Since Point cloudsor meshes are not in a regular format, most researcherstypically transform such data to regular 3D voxel grids orcollections of images ( , views) before feeding them toa deep net architecture. This data representation transfor-mation, however, renders the resulting data unnecessarilyvoluminous while also introducing quantization artifactsthat can obscure natural invariances of the this reason we focus on a different input rep-resentation for 3D geometry using simply Point clouds and name our resulting deep netsPointNets. Pointclouds are simple and unified structures that avoid thecombinatorial irregularities and complexities of meshes,and thus are easier to learn from. The PointNet, however,* indicates equal SegmentationPointNetSemantic SegmentationInput Point Cloud ( Point set representation)Figure of propose a novel deep netarchitecture that consumes raw Point cloud (set of points) withoutvoxelization or rendering.

3 It is a unified architecture that learnsboth global and local Point features, providing a simple, efficientand effective approach for a number of 3D recognition has to respect the fact that a Point cloud is just aset of points and therefore invariant to permutations of itsmembers, necessitating certain symmetrizations in the netcomputation. Further invariances to rigid motions also needto be PointNet is a unified architecture that directlytakes Point clouds as input and outputs either class labelsfor the entire input or per Point segment/part labels foreach Point of the input. The basic architecture of ournetwork is surprisingly simple as in the initial stages eachpoint is processed identically and independently. In thebasic setting each Point is represented by just its threecoordinates(x, y, z). Additional dimensions may be addedby computing normals and other local or global to our approach is the use of a single symmetricfunction, max pooling. Effectively the network learns aset of optimization functions/criteria that select interestingor informative points of the Point cloud and encode thereason for their selection.

4 The final fully connected layersof the network aggregate these learnt optimal values into theglobal descriptor for the entire shape as mentioned above(shape classification) or are used to predict per Point labels(shape segmentation).Our input format is easy to apply rigid or affine transfor-mations to, as each Point transforms independently. Thuswe can add a data-dependent spatial transformer networkthat attempts to canonicalize the data before the PointNetprocesses them, so as to further improve the provide both a theoretical analysis and an ex-perimental evaluation of our approach. We show thatour network can approximate any set function that iscontinuous. More interestingly, it turns out that our networklearns to summarize an input Point cloud by a sparse set ofkey points, which roughly corresponds to the skeleton ofobjects according to visualization. The theoretical analysisprovides an understanding why our PointNet is highlyrobust to small perturbation of input points as well asto corruption through Point insertion (outliers) or deletion(missing data).

5 On a number of benchmark datasets ranging from shapeclassification, part segmentation to scene segmentation,we experimentally compare our PointNet with state-of-the-art approaches based upon multi-view and volumetricrepresentations. Under a unified architecture, not only isour PointNet much faster in speed, but it also exhibits strongperformance on par or even better than state of the key contributions of our work are as follows: We design a novel deep net architecture suitable forconsuming unordered Point sets in 3D; We show how such a net can be trained to perform3D shape classification, shape part segmentation andscene semantic parsing tasks; We provide thorough empirical and theoretical analy-sis on the stability and efficiency of our method; We illustrate the 3D features computed by the selectedneurons in the net and develop intuitive explanationsfor its problem of processing unordered sets by neural netsis a very general and fundamental problem we expect thatour ideas can be transferred to other domains as Related WorkPoint Cloud FeaturesMost existing features for pointcloud are handcrafted towards specific tasks.

6 Point featuresoften encode certain statistical properties of points and aredesigned to be invariant to certain transformations, whichare typically classified as intrinsic [2,21,3] or extrinsic[18,17,13,10,5]. They can also be categorized as localfeatures and global features. For a specific task, it is nottrivial to find the optimal feature Learning on 3D Data3D data has multiple popularrepresentations, leading to various approaches for CNNs:[25,15,16] are the pioneers applying3D convolutional neural networks on voxelized , volumetric representation is constrained by itsresolution due to data sparsity and computation cost of3D convolution. FPNN [12] and Vote3D [23] proposedspecial methods to deal with the sparsity problem; however,their operations are still on sparse volumes, it s challengingfor them to process very large Point :[20,16] have tried to render 3D Point cloud orshapes into 2D images and then apply 2D conv nets toclassify them. With well engineered image CNNs, thisline of methods have achieved dominating performance onshape classification and retrieval tasks [19].

7 However, it snontrivial to extend them to scene understanding or other3D tasks such as Point classification and shape CNNs:Some latest works [4,14] use spectralCNNs on meshes. However, these methods are currentlyconstrained on manifold meshes such as organic objectsand it s not obvious how to extend them to non-isometricshapes such as DNNs:[6,8]firstly convert the 3D data into a vector, by extractingtraditional shape features and then use a fully connected netto classify the shape. We think they are constrained by therepresentation power of the features Learning on Unordered SetsFrom a data structurepoint of view, a Point cloud is an unordered set of most works in deep Learning focus on regular inputrepresentations like sequences (in speech and languageprocessing), images and volumes (video or 3D data), notmuch work has been done in deep Learning on Point recent work from Oriol Vinyals et al [22] looksinto this problem. They use a read-process-write networkwith attention mechanism to consume unordered input setsand show that their network has the ability to sort , since their work focuses on generic sets and NLPapplications, there lacks the role of geometry in the Problem StatementWe design a deep Learning framework that directlyconsumes unordered Point sets as inputs.

8 A Point cloud isrepresented as a set of 3D points{Pi|i= 1, .., n}, whereeach pointPiis a vector of its(x, y, z)coordinate plus extrafeature channels such as color, normal etc. For simplicityand clarity, unless otherwise noted, we only use the(x, y, z)coordinate as our Point s the object classification task, the input Point cloud iseither directly sampled from a shape or pre-segmented froma scene Point cloud. Our proposed deep network outputskscores for all thekcandidate classes. For semanticsegmentation, the input can be a single object for part regionsegmentation, or a sub-volume from a 3D scene for objectregion segmentation. Our model will outputn mscoresfor each of thenpoints and each of themsemantic pointspoint featuresoutput scoresmaxpoolshared shared shared nx3nx3nx64nx64nx10241024n x 1088nx128mlp (64,64)mlp (64,128,1024)inputtransformfeaturetransf ormmlp(512,256,k)global featuremlp (512,256,128)T-Netmatrixmultiply3x3trans formT-Netmatrixmultiply64x64transformsha red mlp (128,m)output scoresnxmkClassification NetworkSegmentation NetworkFigure classification network takesnpoints as input, applies input and feature transformations, and thenaggregates Point features by max pooling.

9 The output is classification scores forkclasses. The segmentation network is an extension to theclassification net. It concatenates global and local features and outputs per Point scores. mlp stands for multi-layer perceptron, numbersin bracket are layer sizes. Batchnorm is used for all layers with ReLU. Dropout layers are used for the last mlp in classification Deep Learning on Point SetsThe architecture of our network ( ) is inspired bythe properties of Point sets inRn( ). Properties of Point Sets inRnOur input is a subset of points from an Euclidean has three main properties: Unordered. Unlike pixel arrays in images or voxelarrays in volumetric grids, Point cloud is a set of pointswithout specific order. In other words, a network thatconsumesN3D Point sets needs to be invariant toN!permutations of the input set in data feeding order. Interaction among points. The points are from a spacewith a distance metric. It means that points are notisolated, and neighboring points form a meaningfulsubset.

10 Therefore, the model needs to be able tocapture local structures from nearby points, and thecombinatorial interactions among local structures. Invariance under transformations. As a geometricobject, the learned representation of the Point setshould be invariant to certain transformations. Forexample, rotating and translating points all togethershould not modify the global Point cloud category northe segmentation of the PointNet ArchitectureOur full network architecture is visualized in Fig2,where the classification network and the segmentationnetwork share a great portion of structures. Please read thecaption of Fig2for the network has three key modules: the max poolinglayer as a symmetric function to aggregate information fromall the points, a local and global information combinationstructure, and two joint alignment networks that align bothinput points and Point will discuss our reason behind these design choicesin separate paragraphs Function for Unordered InputIn orderto make a model invariant to input permutation, threestrategies exist: 1) sort input into a canonical order; 2) treatthe input as a sequence to train an RNN, but augment thetraining data by all kinds of permutations; 3) use a simplesymmetric function to aggregate the information from eachpoint.