Image Colorization with Deep Convolutional Neural …

1 Image Colorization with deep Convolutional Neural NetworksJeff present a Convolutional - Neural - network -based sys-tem that faithfully colorizes black and white photographicimages without direct human assistance. We explore var-ious network architectures, objectives, color spaces, andproblem formulations. The final classification-based modelwe build generates colorized images that are significantlymore aesthetically-pleasing than those created by the base-line regression-based model, demonstrating the viability ofour methodology and revealing promising avenues for fu-ture IntroductionAutomated Colorization of black and white images hasbeen subject to much research within the computer visionand machine learning communities.

2 Beyond simply beingfascinating from an aesthetics and artificial intelligence per-spective, such capability has broad practical applicationsranging from video restoration to Image enhancement forimproved , we take a statistical-learning-driven approach to-wards solving this problem. We design and build a convolu-tional Neural network (CNN) that accepts a black-and-whiteimage as an input and generates a colorized version of theimage as its output; Figure 1 shows an example of such apair of input and output images. The system generates itsoutput based solely on images it has learned from in thepast, with no further human recent years, CNNs have emerged as the de facto stan-dard for solving Image classification problems, achievingerror rates lower than4%in the ImageNet challenge [12].

3 CNNs owe much of their success to their ability to learnand discern colors, patterns, and shapes within images andassociate them with object classes. We believe that thesecharacteristics naturally lend themselves well to colorizingimages since object classes, patterns, and shapes generallycorrelate with color 1. Sample input Image (left) and output Image (right).2. Related workOur project was inspired in part by Ryan Dahl s CNN-based system for automatically colorizing images [2].Dahl s system relies on several ImageNet-trained layersfrom VGG16 [13], integrating them with an autoencoder-like system with residual connections that merge interme-diate outputs produced by the encoding portion of the net-work comprising the VGG16 layers with those producedby the latter decoding portion of the network .

4 The resid-ual connections are inspired by those existing in the ResNetsystem built by He et al that won the 2015 ImageNet chal-lenge [5]. Since the connections link downstream networkedges with upstream network edges, they purportedly allowfor more rapid propagation of gradients through the system,which reduces training convergence time and enables train-ing deeper networks more reliably. Indeed, Dahl reportsmuch larger decreases in training loss on each training iter-ation with his most recent system compared with an earliervariant that did not utilize residual terms of results, Dahl s system performs extremelywell in realistically colorizing foliage, skies, and skin.

5 We,however, notice that in numerous cases, the images gen-erated by the system are predominantly sepia-toned andmuted in color. We note that Dahl formulates Image col-orization as a regression problem wherein the training ob-jective to be minimized is a sum of Euclidean distances be-tween each pixel s blurred color channel values in the targetimage and predicted Image . Although regression does seemto be well-suited to the task due to the continuous natureof color spaces, in practice, a classification-based approachmay work better. To understand why, consider a pixel that1exists in a flower petal across multiple images that are iden-tical, save for the color of the flower petals.

6 Depending onthe picture, this pixel can take on various tones of red, yel-low, blue, and more. with a regression-based system thatuses an`2loss function, the predicted pixel value that min-imizes the loss for this particular pixel is the mean pixelvalue. Accordingly, the predicted pixel ends up being anunattractive, subdued mixture of the possible colors. Gener-alizing this scenario, we hypothesize that a regression-basedsystem would tend to generate images that are desaturatedand impure in color tonality, particularly for objects thattake on many colors in the real world, which may explainthe lack of punchiness in color in the sample images col-orized by Dahl s ApproachWe build a learning pipeline that comprises a Neural net-work and an Image pre-processing General pipelineDuring training time, our program reads images of pixeldimension224 224and3channels corresponding to red,green, and blue in the RGB color space.

7 The images areconverted toCIELUV color space. The black and whiteluminanceLchannel is fed to the model as input. TheUandVchannels are extracted as the target test time, the model accepts a224 224 1blackand white Image . It generates two arrays, each of dimension224 224 1, corresponding to theUandVchannelsof theCIELUV color space. The three channels are thenconcatenated together to form theCIELUV representationof the predicted Transfer learningWe initialized parts of model with a VGG16 instance thathas been pretrained on the ImageNet dataset. Since imagesubject matter often implies color palette, we reason thata network that has demonstrated prowess in discriminatingamongst the many classes present in the ImageNet datasetwould serve well as the basis for our network .

8 This moti-vates our decision to apply transfer learning in this Activation functionWe use the rectified linear unit as the nonlinearity thatfollows each of our Convolutional and dense layers. Mathe-matically, the rectified linear unit is defined asf(x) = max(0,x)The rectified linear unit has been empirically shown togreatly accelerate training convergence [9]. Moreover, it ismuch simpler to compute than many other conventional ac-tivation functions. For these reasons, the rectified linear unithas become standard for Convolutional Neural 2. Regression network downside of using the rectified linear unit as the ac-tivation function in a Neural network is that the model pa-rameters can be updated in such a way that the function sactive region is always in the zero-gradient section.

9 In thisscenario, subsequent backpropagated gradients will alwaysbe zero, hence rendering the corresponding neurons perma-nently inactive. In practice, this has not been an issue Batch normalizationIoffe et al introduced batch normalization as a means ofdramatically reducing training convergence time and im-proving accuracy [7]. For our networks, we place a batchnormalization layer before every non-linearity layer apartfrom the last few layers before the output. In our trials, wehave found that doing so does improve the training rate ofthe Baseline regression modelWe used a regression-based model similar to the modeldescribed in [2] as our baseline.

10 Figure 2 shows the struc-ture of this baseline describe this architecture as comprising a summa-rizing , encoding process on the left side followed by a creating , decoding process on the right sideThe architecture of the leftmost column of layers is in-herited from a portion of the VGG16 network . During this summarizing process, the size (height and width) of the2feature map shrinks while the depth increases. As the modelforwards its input deeper into the network , it learns a richcollection of higher-order abstract featuresThe creating process on the right column is a modifiedversion of the residual encoder structure described in [2].