Detecting Unexpected Obstacles for Self-Driving Cars ...

1 Detecting Unexpected Obstacles for Self-Driving Cars:Fusing Deep Learning and Geometric ModelingSebastian Ramos1,2, Stefan Gehrig1, Peter Pinggera1, Uwe Franke1and Carsten Rother2 Abstract The detection of small road hazards, such as lostcargo, is a vital capability for Self-Driving cars. We tackle thischallenging and rarely addressed problem with a vision systemthat leverages appearance, contextual as well as geometric utilize the appearance and contextual cues, we proposea new deep learning-based obstacle detection framework. Herea variant of a fully convolutional network is used to predicta pixel-wise semantic labeling of(i) free-space,(ii) on-roadunexpected Obstacles , and(iii) background. The geometric cuesare exploited using a state-of-the-art detection approach thatpredicts Obstacles from stereo input images via model-basedstatistical hypothesis present a principled Bayesian framework to fuse thesemantic and stereo-based detection results.

2 The mid-levelStixel representation is used to describe Obstacles in a flexible,compact and robust evaluate our new obstacle detection system on theLostand Founddataset, which includes very challenging sceneswith Obstacles of only 5 cm height. Overall, we report amajor improvement over the state-of-the-art, with relativeperformance gains of up to 50%. In particular, we achievea detection rate of over 90% for distances of up to 50 m. Oursystem operates at 22 Hz on our Self-Driving INTRODUCTIONThe detection of small-sized and Unexpected road hazards,such as lost cargo, is a very demanding and important taskfor human drivers. US traffic reports show that approximately150 people are killed annually due to lost cargo on the road[2]. Consequently, Self-Driving cars operating on public roadshave to be able to avoid running over such Obstacles by allmeans.

3 Considering the resulting requirements on environ-ment perception systems, this task represents a challengingand so far largely unexplored research topic. In order to closethe gap between semi-autonomous and fully autonomousdriving capabilities, a robust and accurate solution to thisproblem has to be rather expensive high-end time-of-flight sensors,RGB cameras represent a promising means to approachthis task due to their high resolution. Previous systems thataddressed this problem make use of stereo cameras in orderto take advantage of geometric cues for the detection and 3 Dlocalization of small generic Obstacles [1].However, safe operation of fully autonomous vehiclesrequires an extremely high degree of accuracy and reliabil-ity with regard to environment perception systems. In theparticular case of Detecting objects that cover only small1S.

4 Ramos, S. Gehrig, P. Pinggera and U. Franke are with the En-vironment Perception Dept., Daimler AG R&D, Sindelfingen, Rother is with the Computer Vision Lab Dresden, TU Dresden,Dresden, image and close-up showing four small but critical semantic labeling result of our appearance-based representation of the detection results of the probabilistic fusionsystem (semantics + geometry).Fig. scene from theLost and Founddataset [1] and corre-sponding results of the proposed system. All relevant Obstacles are detectedsuccessfully despite their challenging size and appearance. Semantic labelsare color-coded by class, final detections are color-coded by regions and occur in all possible shapes and dimen-sions, utilizing geometric cues alone is not sufficient. Heremachine learning techniques that leverage appearance cuesrepresent a promising complementary approach to the currentgeometry-only systems.

5 However, due to the high variabilityin appearance of both objects and road surfaces, traditionalsupervised learning systems are bound to encounter problemsin , research on artificial intelligence has experi-enced a substantial boost due to advances in deep learningmethods [3]. In the field of computer vision, deep neural net-works and especially deep Convolutional Neural Networks(CNNs) have emerged as an extremely powerful tool to learnfeature representations for a large variety of tasks from low-level (e. g. image segmentation) to high-level vision (e. detection and classification). Arguably, an even greaterstrength of CNNs lies in their ability to exploit contextualinformation learned from large quantities of training our application, small Obstacles lying on the road [ ] 20 Dec 2016one common property, which is context, that can be exploiteddespite the inherent variance in shape, size and particular, such Obstacles are generally surrounded bydrivable road area and thus stand out as an anomaly from anotherwise regular or piece-wise homogeneous this paper, we introduce a new deep learning-basedapproach to detect generic, small, and Unexpected obstacleson the road ahead.

6 In this way we demonstrate how theability of CNNs to learn context and generalize informationfrom training data can overcome one of their main openproblems: handling outliers and the open world . This refersto the tasks of correctly handling novel, previously unseenobjects as well as modeling a suitable background , we present a probabilistic fusion approachthat combines our learning-based detection method with thecurrently best performing stereo-based system for this our in-depth evaluation we report a relative increase indetection rate by 50% compared to the state-of-the-art, witha simultaneous decrease of 13% in false remainder of this paper is organized as follows: Sec-tion II reviews related work on the considered topic, coveringgeometric model-based as well as machine learning meth-ods. In Section III the proposed detection approaches aredescribed in detail.

7 Section IV offers an in-depth evaluation,focusing on a quantitative analysis of detection performanceand false positive rates, followed by qualitative results aswell as discussions. The final section provides a summaryand a brief RELATEDWORKO bstacle detection has been a heavily researched topic fordecades, spanning a variety of application areas. However,little work has focused on the detection of small, genericand Unexpected Obstacles in driving scenarios. We mainlyconsider camera-based methods for the detection and local-ization of such generic Obstacles in 3D space, using stereocamera setups with small baselines (i. e. less than 25 cm).Classical geometry-based obstacle detection schemes aremostly based on the flat-world-assumption, modeling free-space or ground as a single planar surface and characterizingobstacles by their height-over-ground ( [4]).

8 Geometricdeviations from this reference plane are often estimated ei-ther from image data directly [5] or via mode extraction fromthe v-disparity histogram on multiple scales [6]. In order tocope with deviations from the flat-world-assumption, moresophisticated ground profile models have been introduced, piece-wise planar longitudinal profiles [7] or splines [8].A recent survey [9] presents an overview of several stereo-based generic obstacle detection approaches that have provento perform well in practice. The methods are grouped intodifferent obstacle representation categories including Stixels[10], Digital Elevation Maps (DEM) [11] and geometricpoint clusters [12]. DEMs have been recently used in theagricultural field for obstacle detection, exploiting a set ofrules per grid cell to determine obstacle candidates [13]. Weselect the Stixel method [10] to serve as one of our baselinesduring our experimental evaluation.

9 The Stixel algorithmdiscriminates between an estimated global ground surfacemodel and a set of rectangular vertical obstacle segments,providing an efficient and robust representation of the methods presented above all rely on precomputedstereo disparity maps and obtain reliable detection resultsbased on generic geometric criteria. They perform well fordetecting medium-sized objects at close to medium range,with reduced performance for longer ranges and present work utilizes the obstacle detection approachof [1] as one of the main system components. This methodwas originally presented in [14] within the scope of high-sensitivity long range detection tasks. It was extended in[1] and successfully applied to the detection of small andchallenging road hazards. The stereo-based detection systemperforms independent, local tests, comparing obstacle againstfree-space hypotheses using constrained plane models, whichare optimized directly on the underlying image data.

10 Themethod however does require a certain minimum obstacleheight in the image (i. e. in pixels), limiting the detectionrange for very small Obstacles with the given camera setupto less than 40 m. Additionally, theLost and Founddatasetwas introduced in this appearance-based detection of Unexpected obstacleshas not been widely investigated. To the best of our knowl-edge, the recent work of [15] first explored this particularproblem from a machine learning perpective. This workfocuses on the detection of regions with anomalous appear-ance on the road using a Restricted Boltzmann Machineneural network. This network is trained to reconstruct theappearance of the road in a given scenario. By subtractionof the observed and the reconstructed road appearances, theproposed method is able to segment small generic objectson the road. As mentioned by the authors, this work is at anearly stage and has limitations to generalize to roads withappearances differing from the training examples, a maindrawback for the considered application not directly focused on the detection of un-expected road hazards, several works have been exploringmachine/deep learning techniques for the segmentation ofthe free (drivable) space from visual data in the context ofautonomous vehicle navigation.