Pseudo-LiDAR From Visual Depth Estimation: Bridging the ...

1 Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous DrivingYan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q. WeinbergerCornell University, Ithaca, NY{yw763, wc635, dg595, bh497, mc288, object detection is an essential task in autonomousdriving. Recent techniques excel with highly accurate de-tection rates, provided the 3D input data is obtained fromprecise but expensive lidar technology. Approaches basedon cheaper monocular or stereo imagery data have, untilnow, resulted in drastically lower accuracies a gap thatis commonly attributed to poor image-based Depth estima-tion. However, in this paper we argue that it is not the qual-ity of the data but its representation that accounts for themajority of the difference. Taking the inner workings of con-volutional neural networks into consideration, we proposeto convert image-based Depth maps to Pseudo-LiDAR repre-sentations essentially mimicking the lidar signal.}

2 Withthis representation we can apply different existing lidar -based detection algorithms. On the popular KITTI bench-mark, our approach achieves impressive improvements overthe existing state-of-the-art in image-based performance raising the detection accuracy of objects within the 30mrange from the previous state-of-the-art of 22% to an un-precedented 74%. At the time of submission our algorithmholds the highest entry on the KITTI 3D object detectionleaderboard for stereo-image-based IntroductionReliable and robust 3D object detection is one of the fun-damental requirements for autonomous driving. After all, inorder to avoid collisions with pedestrians, cyclist, and cars,a vehicle must be able to detect them in the first algorithms largely rely on lidar (Light Detec-tion And Ranging), which provide accurate 3D point cloudsof the surrounding environment. Although highly precise,alternatives to lidar are desirable for multiple , lidar is expensive, which incurs a hefty premiumfor autonomous driving hardware.

3 Second, over-reliance ona single sensor is an inherent safety risk and it would beadvantageous to have a secondary sensor to fall-back ontoInputDepth MapPseudo- lidar (Bird s-eye) View)Figure 1: Pseudo-LiDAR signal from Visual Depth : a KITTI street scene with super-imposedbounding boxes around cars obtained with lidar (red) andpseudo- lidar (green). Bottom-left: estimated disparitymap. Right: Pseudo-LiDAR (blue) vs. lidar (yellow) the Pseudo-LiDAR points align remarkably well with theLiDAR ones. Best viewed in color (zoom in for details.)in case of an outage. A natural candidate are images fromstereo or monocular cameras. Optical cameras are highlyaffordable (several orders of magnitude cheaper than Li-DAR), operate at a high frame rate, and provide a densedepth map rather than the 64 or 128 sparse rotating laserbeams that lidar signal is inherently limited recent publications have explored the use ofmonocular and stereo Depth (disparity) estimation [13,19,32] for 3D object detection [5,6,22,30].

4 However, to-datethe main successes have been primarily in supplementingLiDAR approaches. For example, one of the leading algo-rithms [17] on the KITTI benchmark [11,12] uses sensorfusion to improve the 3D average precision (AP) for carsfrom 66% for lidar to 73% with lidar and monocularimages. In contrast, among algorithms that use only images,the state-of-the-art achieves a mere 10% AP [30].One intuitive and popular explanation for such inferiorperformance is the poor precision of image-based Depth es-timation. In contrast to lidar , the error of stereo Depth es-timation grows quadratically with Depth . However, a visualcomparison of the 3D point clouds generated by lidar anda state-of-the-art stereo Depth estimator [3] reveals a highquality match (cf. ) between the two data modalities even for faraway this paper we provide an alternative explanation withsignificant performance implications. We posit that themajor cause for the performance gap between stereo andLiDAR is not a discrepancy in Depth accuracy, but apoor choice of representations of the 3D information forConvNet-based 3D object detection systems operating onstereo.

5 Specifically, the lidar signal is commonly rep-resented as 3D point clouds [23] or viewed from the top-down bird s-eye view perspective [33], and processed ac-cordingly. In both cases, the object shapes and sizes are in-variant to Depth . In contrast, image-based Depth is denselyestimated for each pixel and often represented as additionalimage channels [6,22,30], making far-away objects smallerand harder to detect. Even worse, pixel neighborhoods inthis representation group together points from far-away re-gions of 3D space. This makes it hard for convolutionalnetworks relying on 2D convolutions on these channels toreason about and precisely localize objects in evaluate our claim, we introduce a two-step approachfor stereo-based 3D object detection. We first convert theestimated Depth map from stereo or monocular imagery intoa 3D point cloud, which we refer to aspseudo-LiDARas itmimics the lidar signal. We then take advantage of ex-isting lidar -based 3D object detection pipelines [16,23],which we train directly on the Pseudo-LiDAR representa-tion.

6 By changing the 3D Depth representation to Pseudo-LiDAR we obtain an unprecedented increase in accuracyof image-based 3D object detection algorithms. Specifi-cally, on the KITTI benchmark with IoU (intersection-over-union) at for moderately hard car instances themetric used in the official leaderboard we achieve 3D AP on the validation set: almost a 350% im-provement over the previous state-of-the-art image-basedapproach. Furthermore, we halve the gap between stereo-based and lidar -based evaluate multiple combinations of stereo Depth es-timation and 3D object detection algorithms and arrive atremarkably consistent results. This suggests that the gainswe observe are because of thepseudo-LiDARrepresentationand areless dependenton innovations in 3D object detec-tion architectures or Depth estimation sum, the contributions of the paper are two-fold. First,we show empirically that a major cause for the performancegap between stereo-based and lidar -based 3D object de-tection is not the quality of the estimated Depth but itsrepre-sentation.

7 Second, we proposepseudo-LiDARas a new rec-ommended representation of estimated Depth for 3D objectdetection and show that it leads to state-of-the-art stereo-based 3D object detection, effectivelytriplingprior art. Ourresults point towards the possibility of using stereo camerasin self-driving cars potentially yielding substantial costreductions and/or safety Related WorkLiDAR-based 3D object work is inspiredby the recent progress in 3D vision and lidar -based 3 Dobject detection. Many recent techniques use the fact thatLiDAR is naturally represented as 3D point clouds. Forexample, frustum PointNet [23] applies PointNet [24] toeach frustum proposal from a 2D object detection [7] projects lidar points into both bird-eye view(BEV) and frontal view to obtain multi-view features. Vox-elNet [34] encodes 3D points into voxels and extracts fea-tures by 3D convolutions. UberATG-ContFuse [17], one ofthe leading algorithms on the KITTI benchmark [12], per-forms continuous convolutions [27] to fuse Visual and BEVLiDAR features.

8 All these algorithms assume that the pre-cise 3D point coordinates are given. The main challengethere is thus on predicting point labels or drawing boundingboxes in 3D to locate and monocular-based Depth keyingredient for image-based 3D object detection methods is areliable Depth estimation approach to replace lidar . Thesecan be obtained through monocular [10,13] or stereo vi-sion [3,19]. The accuracy of these systems has increaseddramatically since early work on monocular Depth estima-tion [8,15,26]. Recent algorithms like DORN [10] com-bine multi-scale features with ordinal regression to predictpixel Depth with remarkably low errors. For stereo vision,PSMNet [3] applies Siamese networks for disparity estima-tion, followed by 3D convolutions for refinement, resultingin an outlier rate less than2%. Recent work has made thesemethods mode efficient [28], enabling accurate disparity es-timation to run at 30 FPS on mobile 3D object rapid progress onstereo and monocular Depth estimation suggests that theycould be used as a substitute for lidar in image-based3D object detection algorithms.

9 Existing algorithms of thisflavor are largely built upon 2D object detection [25], im-posing extra geometric constraints [2,4,21,29] to create3D proposals. [5,6,22,30] apply stereo-based Depth es-timation to obtain the true 3D coordinates of each 3D coordinates are either entered as additional in-put channels into a 2D detection pipeline, or used to extracthand-crafted features. Although these methods have maderemarkable progress, the state-of-the-art for 3D object de-tection performance lags behind lidar -based methods. Aswe discuss in Section3, this might be because of the depthrepresentation used by these ApproachDespite the many advantages of image-based 3D objectrecognition, there remains a glaring gap between the state-of-the-art detection rates of image and lidar -based ap-8446 Stereo/MonodepthLiDAR-baseddetectionSter eo/Mono imagesDepth estimationDepth mapPseudo LiDAR3D object detectionPredicted 3D boxesFigure 2:The proposed pipeline for image-based 3D object stereo or monocular images, we first predictthe Depth map, followed by back-projecting it into a 3D point cloud in the lidar coordinate system .

10 We refer to thisrepresentation aspseudo- lidar , and process it exactly like lidar any lidar -based detection algorithms can be (see Table1in ). It is tempting toattribute this gap to the obvious physical differences andits implications between lidar and camera example, the error of stereo-based 3D Depth estimationgrows quadratically with the Depth of an object, whereasfor Time-of-Flight (ToF) approaches, such as lidar , thisrelationship is approximately some of these physical differences do likelycontribute to the accuracy gap, in this paper we claim that alarge portion of the discrepancy can be explained by the datarepresentation rather than its quality or underlying physicalproperties associated with data fact, recent algorithms for stereo Depth estimation cangenerate surprisingly accurate Depth maps [3] (see figure1).Our approach to close the gap is therefore to carefully re-move the differences between the two data modalities andalign the two recognition pipelines as much as possible.