Depth Prediction Without the Sensors: Leveraging Structure ...

1 Depth Prediction Without the Sensors: Leveraging Structure for UnsupervisedLearning from Monocular VideosVincent Casser 1 Soeren PirkReza Mahjourian2 Anelia AngelovaGoogle Brain1 Institute for Applied Computational Science, Harvard University; Google Brain2 University of Texas at Austin; Google rezama, to predict scene Depth from RGB inputs is a chal-lenging task both for indoor and outdoor robot this work we address unsupervised learning of scenedepth and robot ego-motion where supervision is providedby monocular videos, as cameras are the cheapest, least re-strictive and most ubiquitous sensor for work in unsupervised image-to- Depth learning hasestablished strong baselines in the domain.

2 We propose anovel approach which produces higher quality results, is ableto model moving objects and is shown to transfer across datadomains, from outdoors to indoor scenes. The main ideais to introduce geometric Structure in the learning process,by modeling the scene and the individual objects; cameraego-motion and object motions are learned from monocularvideos as input. Furthermore an online refinement method isintroduced to adapt learning on the fly to unknown proposed approach outperforms all state-of-the-art ap-proaches, including those that handle motion throughlearned flow. Our results are comparable in quality to the oneswhich used stereo as supervision and significantly improvedepth Prediction on scenes and datasets which contain a lotof object motion.

3 The approach is of practical relevance, as itallows transfer across environments, by transferring modelstrained on data collected for robot navigation in urban scenesto indoor navigation settings. The code associated with thispaper can be found scene Depth from input imagery is important forrobot navigation, both for indoors and outdoors settings. Su-pervised dense Depth Prediction per single image has beenvery successful with deep neural networks (Eigen, Puhrsch,and Fergus 2014; Laina et al. 2016; Wang, Fouhey, andGupta 2015; Li, Klein, and Yao 2017), where learned mod-els convincingly outperform those with hand-crafted fea-tures (Ladicky, Zeisl, and Pollefeys 2014; Karsch, Liu, andKang 2014a).

4 However, supervised learning of scene depthrequires expensive Depth sensors which may not be read-ily available in most robotics scenarios and may introducetheir own sensor noise. To that end a number of unsupervisedimage-to- Depth methods have been proposed, which demon-strate that unsupervised Depth Prediction models are more Work done while at Google 2019, Association for the Advancement of ArtificialIntelligence ( ). All rights 1: Problem setup: Obtaining scene Depth predictionfrom RGB image input. Training is unsupervised and frommonocular videos only. No Depth sensor supervision is than sensor-supervised ones (Zhou et al. 2017;Garg, Carneiro, and Reid 2016), predominantly due to issueswith sensor readings, missing or noisy sensor research led to a number of improvements in which un-supervised methods have decreased Prediction errors signif-icantly, including methods that use stereo (Godard, Aodha,and Brostow 2017), or independently trained optical flowmodels during learning (Wang et al.)

5 2018).We propose a novel approach that explicitlymodels3D motionsof moving objects, together with camera ego-motion, and adapts to new environments by learning withanonline refinementof multiple frames. With a principledway of handling motion and a newly introduced object sizeconstraint, we are the first to effectively learn from highlydynamic scenes in a monocular setting. Our approach intro-duces Structure in the learning process by representing ob-jects in 3D and modeling motion as SE3 transforms; this isdone by fully differentiable operations and is trained fromuncalibrated monocular videos. Extensive experiments areconducted on two publicly available datasets.

6 Our algorithmyields significant improvements on both datasets and on bothdepth and ego-motion estimation, compared to the state-of-the-art; our method is also competitive to models trainedwith stereo. Furthermore, we evaluate direct domain trans-fer, by training on one dataset and testing on another, with-out fine-tuning. We present transfer results across KITTI andCityscapes, as well as, training on Cityscapes and testingon an indoor Robot Navigation dataset. This demonstratesthe method is applicable across domains and that exhaus-tive data-collection for training may not be needed. [ ] 15 Nov 2018proposed algorithm adapts to new environment and learnsto predict Depth and ego-motion online.

7 To the best of ourknowledge, while online-learning is a well-known concept,we are the first to introduce an online refinement method fordomain transfer in this unsupervised learning setting. We donot only show promising results to illustrate this, but also ex-pect the refinement method to be useful in better leveragingtemporally and spatially related content during continuousinference. While using online refinement requires additionalcomputation, our main motion model runs at 50 FPS and30 FPS on a Geforce 1080Ti for batch 4 and 1, respectively,making it real-time capable on several state-of-the-art WorkScene Depth estimation has been a long standing problem invision and robotics.

8 Numerous approaches, involving stereoor multi-view Depth estimation exist. Recently a learning-based concept for image-to- Depth estimation has emergedfueled by availability of rich feature representations, learnedfrom raw data (Eigen, Puhrsch, and Fergus 2014; Lainaet al. 2016). These approaches have shown compelling re-sults as compared to traditional methods (Karsch, Liu, andKang 2014b). Pioneering work in unsupervised image-to- Depth learning has been proposed by (Zhou et al. 2017;Garg, Carneiro, and Reid 2016) where no Depth or ego-motion is needed as supervision. Many subsequent workshave improved the initial results in both the monocular set-ting (Yang et al.)

9 2017; Yin 2018) and when using stereo dur-ing training (Godard, Aodha, and Brostow 2017; Ummen-hofer et al. 2017; Zhan et al. 2018; Yang et al. 2018a).However, these methods still fall short in practice becauseobject movements in dynamic scenes are not handled. Inthese highly dynamic scenes, the abovementioned methodstend to fail as they can not explain object motion. To thatend, optical flow models, trained separately, have been usedwith moderate improvements (Yin 2018; Yang et al. 2018b;2018a). Our motion model is most aligned to these meth-ods as we similarly use a pre-trained model, but propose touse the geometric Structure of the scene and model all ob-jects motion including camera ego-motion.

10 The refinementmethod is related to prior work (Bloesch et al. 2018) whouse lower dimensional representations to fuse subsequentframes; our work shows that this can be done in the origi-nal space to a very good MethodThe main learning setup is unsupervised learning of depthand ego-motion from monocular video (Zhou et al. 2017),where the only source of supervision is obtained from thevideo itself. We here propose a novel approach which is ableto model dynamic scenes by modeling object motion, andthat can optionally adapt its learning strategy with an on-line refinement technique. Note that both ideas are tangen-tial and can be used either separately or jointly.