PIXOR: Real-Time 3D Object Detection From Point Clouds

1 PIXOR: Real-Time 3D Object Detection from Point CloudsBin Yang, Wenjie Luo, Raquel UrtasunUber Advanced Technologies GroupUniversity of Toronto{byang10, wenjie, address the problem of Real-Time 3D Object detec-tion from Point Clouds in the context of autonomous driv-ing. speed is critical as Detection is a necessary compo-nent for safety. Existing approaches are, however, expensivein computation due to high dimensionality of Point utilize the 3D data more efficiently by representing thescene from the Bird s Eye View (BEV), and propose PIXOR,a proposal-free, single-stage detector that outputs oriented3D Object estimates decoded from pixel-wise neural net-work predictions. The input representation, network archi-tecture, and model optimization are specially designed tobalance high accuracy and Real-Time efficiency. We validatePIXOR on two datasets: the KITTI BEV Object detectionbenchmark, and a large-scale 3D vehicle Detection bench-mark.}

2 In both datasets we show that the proposed detectorsurpasses other state-of-the-art methods notably in terms ofAverage Precision (AP), while still runs at 10 IntroductionOver the last few years we have seen a plethora of meth-ods that exploit Convolutional Neural Networks to produceaccurate 2D Object detections, typically from a single image[12,11,28,4,27,23]. However, in robotics applicationssuch as autonomous driving we are interested in detectingobjects in 3D space. This is fundamental for motion plan-ning in order to plan a safe approaches to 3D Object Detection exploit differ-ent data sources. Camera based approaches utilize eithermonocular [1] or stereo images [2]. However, accurate 3 Destimation from 2D images is difficult, particularly in longranges. With the popularity of inexpensive RGB-D sen-sors such as Microsoft Kinect, Intel RealSense and ApplePrimeSense, several approaches that utilize depth informa-tion and fuse them with RGB images have been developed[32,33].

3 They have been shown to achieve significant per-formance gains over monocular methods. In the contextof autonomous driving, high-end sensor like lidar (LightDetection And Ranging) is more common because higheraccuracy is needed for safety. The major difficulty in deal-ing with lidar data is that the sensor produces unstruc-tured data in the form of a Point cloud containing typicallyaround1053D points per 360-degree sweep. This poses alarge computational challenge for modern forms of Point cloud representation have beenexplored in the context of 3D Object Detection . The mainidea is to form a structured representation where standardconvolution operation can be applied. Existing representa-tions are mainly divided into two types: 3D voxel grids and2D projections. A 3D voxel grid transforms the Point cloudinto a regularly spaced 3D grid, where each voxel cell cancontain a scalar value ( , occupancy) or vector data ( ,hand-crafted statistics computed from the points within thatvoxel cell).

4 3D convolution is typically applied to extracthigh-order representation from the voxel grid [6]. However,since Point Clouds are sparse by nature, the voxel grid isvery sparse and therefore a large proportion of computationis redundant and unnecessary. As a result, typical systems[6,37,20] only run at 1-2 alternative is to project the Point cloud onto a plane,which is then discretized into a 2D image based representa-tion where 2D convolutions are applied. During discretiza-tion, hand-crafted features (or statistics) are computed aspixel values of the 2D image [3]. Commonly used projec-tions are range view ( , 360-degree panoramic view) andbird s eye view ( , top-down view). These 2D projectionbased representations are more compact, but they bring in-formation loss during projection and discretization. For ex-ample, range-view projection will have distorted Object sizeand shape.

5 To alleviate the information loss, MV3D [3] pro-poses to fuse the 2D projections with the camera image tobring additional information. However, the fused model hasnearly linear computation cost with respect to the number ofinput modals, making Real-Time application this paper, we propose an accuratereal-time3D objectdetector, which we callPIXOR(ORiented 3D Object de-tection fromPIXel-wise neural network predictions), thatoperates on 3D Point Clouds . PIXOR is a single-stage, representation3D lidar Point cloudPixar detector3D BEV detectionsPIXORF igure 1. Overview of the proposed 3D Object detector from Bird s Eye View (BEV) of lidar Point dense Object detector that exploits the 2 DBird s Eye View (BEV) representation in an efficient choose the BEV representation as it is computationallyfriendly compared with 3D voxel grids, and also preservesthe metric space which allows our model to explore priorsabout the size and shape of the Object categories.

6 Our detec-tor outputs accurate oriented bounding boxes in real-worlddimension in bird s eye view. Note that these are 3D esti-mates as we assume that the objects are on the ground. Thisis a reasonable assumption in the autonomous driving sce-nario as vehicles do not demonstrate the effectiveness of our approach in twodatasets, the public KITTI benchmark [10] and a large-scale 3D vehicle Detection dataset (ATG4D). Specifically,PIXOR achieves the highest Average Precision (AP) onKITTI bird s eye view Object Detection benchmark amongall previously published methods, while also runs the fastestamong them (over 10 FPS). We also provide in-depth abla-tion studies on KITTI to investigate how much performancegain each module contributes, and prove the scalability andgeneralization ability of PIXOR by applying it to the large-scale ATG4D Related WorkWe first review recent advances in applying Convolu-tional Neural Networks to Object Detection , and then revisitworks in two related sub-fields, single-stage Object detec-tion and 3D Object CNN-based Object DetectionConvolutional Neural Networks (CNN) have shown out-standing performance in image classification [18].

7 Whenapplied to Object Detection , it is natural to utilize them byrunning inference over cropped regions representing the ob-ject candidates. Overfeat [30] slides a CNN on differentpositions and scales and predicts a bounding box per classat each time. Since the introduction of class-agnostic ob-ject proposals [36,26], proposal based approaches popu-late, with Region-CNN (RCNN) [12] and its faster versions[11,4] being the most seminal work. RCNN first extractsthe whole-image feature map with an ImageNet [5] pre-trained CNN and then predicts a confidence score as wellas box position per proposal via a RoI-pooling operationon the whole-image feature map [13]. Faster-RCNN [28]further proposes to learn to generate region proposals witha CNN and share the feature representation with Detection ,which leads to further gain in both performance and based Object detectors achieve outstanding perfor-mances in many public benchmarks [7,29].

8 However, thetypical two-stage pipeline makes it unsuitable for Single-stage Object DetectionDifferent from the two-stage Detection pipeline that firstpredicts proposals and then refines them, single-stage detec-tors directly predict the finals detections. YOLO [27] andSSD [23] are the most representative works with real-timespeed. YOLO [27] divides the image into sparse grids andmakes multi-class and multi-scale predictions per grid [23] additionally uses pre-defined Object templates (oranchors) to handle large variance in Object size and single-class Object Detection , DenseBox [17] and EAST[38] show that single-stage detector also works well with-out using manually designed anchors. They both adoptthe fully-convolutional network architecture [24] to makedense predictions, where each pixel location corresponds toone Object candidate. Recently RetinaNet [22] shows thatsingle-stage detector can outperform two-stage detector ifclass imbalance problem during training is resolved prop-erly.

9 Our proposed detector follows the idea of single-stagedense Object detector, while further extends these ideas toreal-time 3D Object Detection by re-designing the input rep-resentation, network architecture, and output parameteriza-tion. We also remove the hyper parameter of pre-definedobject anchors by re-defining the objective function of ob-ject localization, which leads to a simpler Detection 3D Object Detection from Point CloudsVote3D [37] uses sliding window on sparse volumes ina 3D voxel grid to detect objects. Hand-crafted geometryfeatures are extracted on each volume and fed into an SVMclassifier [34]. Vote3 Deep [6] also uses the voxel represen-tation of Point Clouds , but extracts features for each volumewith 3D Convolutional Neural Networks [35]. The main is-sue with voxel representations is efficiency, as the 3D voxelgrid usually has high dimensionality.

10 In contrast, VeloFCN[20] projects the 3D Point cloud to front-view and gets a2D depth map. Vehicles are then detected by applying a2D CNN on the depth map. Recently MV3D [3] also usesthe projection representation. It combines CNN features ex-tracted from multiple views (front view, bird s eye view aswell as camera view) to do 3D Object Detection . However,hand-crafted features are computed as the encoding of therasterized images. Our proposed detector, however, uses thebird s eye view representation alone for Real-Time 3D objectdetection in the context of autonomous driving, where weassume that all objects lie on the same PIXOR DetectorIn this paper we propose an efficient 3D Object detectorthat is able to produce very accurate bounding boxes givenLIDAR Point Clouds . Our bounding box estimates not onlycontain the location in 3D space, but also the heading an-gle, since predicting this accurately is very important forautonomous driving.