Deep Multi-Scale Convolutional Neural Network for …

1 Deep Multi-Scale Convolutional Neural Network for Dynamic Scene DeblurringSeungjun NahTae Hyun KimKyoung Mu LeeDepartment of ECE, ASRI, Seoul National University, 151-742, Seoul, Korea{ , blind deblurring for general dynamicscenes is a challenging computer vision problem as blursarise not only from multiple object motions but also fromcamera shake, scene depth variation. To remove thesecomplicated motion blurs, conventional energy optimiza-tion based methods rely on simple assumptions such thatblur kernel is partially uniform or locally linear.}

2 More-over, recent machine learning based methods also dependon synthetic blur datasets generated under these assump-tions. This makes conventional deblurring methods fail toremove blurs where blur kernel is dif cult to approximate orparameterize ( object motion boundaries). In this work,we propose a Multi-Scale Convolutional Neural Network thatrestores sharp images in an end-to-end manner where bluris caused by various sources. Together, we present Multi-Scale loss function that mimics conventional coarse-to- neapproaches. Furthermore, we propose a new large-scaledataset that provides pairs of realistic blurry image and thecorresponding ground truth sharp image that are obtainedby a high-speed camera.

3 With the proposed model trainedon this dataset, we demonstrate empirically that our methodachieves the state-of-the-art performance in dynamic scenedeblurring not only qualitatively, but also IntroductionMotion blur is one of the most commonly arising typesof artifacts when taking photos. Shakes of camera and fastobject motions degrade image quality to undesired blurryimages. Furthermore, various causes such as depth varia-tion, occlusion in motion boundaries make blurs even morecomplex. Single image deblurring problem is to estimatethe unknown sharp image given a blurry image.

4 Earlierstudies focused on removing blurs caused by simple transla-tional or rotational camera motions. More recent works tryto handle general non-uniform blurs caused by depth vari-ation, camera shakes and object motions in dynamic envi-ronments. Most of these approaches are based on followingblur model [28,10,13,11].B=KS+n,(1)whereB,Sandnare vectorized blurry image, latentsharp image, and noise, a large sparsematrix whose rows each contain a local blur kernel actingonSto generate a blurry pixel. In practice, blur kernel isunknown. Thus, blind deblurring methods try to estimatelatent sharp imageSand blur blur kernel for every pixel is a severely ill-posedproblem.

5 Thus, some approaches tried to parametrize blurmodels with simple assumptions on the sources of blurs. In[28,10], they assumed that blur is caused by 3D cameramotion only. However, in dynamic scenes, the kernel es-timation is more challenging as there are multiple movingobjects as well as camera motion. Thus, Kim et al. [14] pro-posed a dynamic scene deblurring method that jointly seg-ments and deblurs a non-uniformly blurred image, allowingthe estimation of complex (non-linear) kernel within a seg-ment. In addition, Kim and Lee [15] approximated the blurkernel to be locally linear and proposed an approach that es-timates both the latent image and the locally linear motionsjointly.

6 However, these blur kernel approximations are stillinaccurate, especially in the cases of abrupt motion discon-tinuities and occlusions. Note that such erroneous kernelestimation directly affects the quality of the latent image,resulting in undesired ringing , CNNs ( Convolutional Neural Networks) havebeen applied in numerous computer vision problems in-cluding deblurring problem and showed promising results[29,25,26,1]. Since no pairs of real blurry image andground truth sharp image are available for supervised learn-ing, they commonly used blurry images generated by con-volving synthetic blur kernels.

7 In [29,25,1], synthesizedblur images with uniform blur kernel are used for , in [26], classification CNN is trained to estimate lo-cally linear blur kernels. Thus, CNN-based models are stillsuited only to some specific types of blurs, and there arerestrictions on more common spatially varying (a)(b)(c)Figure 1. (a) Input blurry image. (b) Result of Sun et al. [26]. (c) Our deblurring result. Our results show clear object boundaries , all the existing methods still have many prob-lems before they could be generalized and used in are mainly due to the use of simple and unrealis-tic blur kernel models.

8 Thus, to solve those problems, inthis work, we propose a novel end-to-end deep learning ap-proach for dynamic scene , we propose a Multi-Scale CNN that directly re-stores latent images without assuming any restricted blurkernel model. Especially, the Multi-Scale architecture isdesigned to mimic conventional coarse-to-fine optimizationmethods. Unlike other approaches, our method does not es-timate explicit blur kernels. Accordingly, our method is freefrom artifacts that arise from kernel estimation errors. Sec-ond, we train the proposed model with a Multi-Scale lossthat is appropriate for coarse-to-fine architecture that en-hances convergence greatly.

9 In addition, we further improvethe results by employing adversarial loss [9]. Third, we pro-pose a new realistic blurry image dataset with ground truthsharp images. To obtain kernel model-free dataset for train-ing, we employ the dataset acquisition method introducedin [17]. As the blurring process can be modeled by the in-tegration of sharp images during shutter time [17,21,16],we captured a sequence of sharp frames of a dynamic scenewith a high-speed camera and averaged them to generate ablurry image by considering gamma training with the proposed dataset and adding properaugmentation, our model can handle general local blur ker-nel implicitly.

10 As the loss term optimizes the result toresemble the ground truth, it even restores occluded re-gions where blur kernel is extremely complex as shown We trained our model with millions of pairs of imagepatches and achieved significant improvements in dynamicscene deblurring. Extensive experimental results demon-strate that the performance of the proposed method is farsuperior to those of the state-of-the-art dynamic scene de-blurring methods in both qualitative and quantitative Related WorksThere are several approaches that employed CNNs fordeblurring [29,26,25,1].