Deep Image Prior - CVF Open Access

1 deep Image PriorDmitry UlyanovSkolkovo Institute of Scienceand Technology, VedaldiUniversity of LempitskySkolkovo Institute of Scienceand Technology convolutional networks have become a populartool for Image generation and restoration. Generally, theirexcellent performance is imputed to their ability to learn re-alistic Image priors from a large number of example this paper, we show that, on the contrary, thestructureofa generator network is sufficient to capture a great deal oflow-level Image statisticsprior to any learning. In orderto do so, we show that a randomly-initialized neural net-work can be used as a handcrafted Prior with excellent re-sults in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same priorcan be used to invert deep neural representations to diag-nose them, and to restore images based on flash-no flashinput from its diverse applications, our approach high-lights the inductive bias captured by standard generatornetwork architectures.

2 It also bridges the gap betweentwo very popular families of Image restoration methods:learning-based methods using deep convolutional networksand learning-free methods based on handcrafted Image pri-ors such as IntroductionDeep convolutional neural networks (ConvNets) cur-rently set the state-of-the-art in inverse Image reconstruc-tion problems such as denoising [5,20] or single-imagesuper-resolution [19,29,18]. ConvNets have also been usedwith great success in more exotic problems such as recon-structing an Image from its activations within certain deepnetworks or from its HOG descriptor [8]. More generally,ConvNets with similar architectures are nowadays used togenerate images using such approaches as generative ad-versarial networks [11], variational autoencoders [16], anddirect pixelwise error minimization [9,3].State-of-the-art ConvNets for Image restoration and gen-Code and supplementary material are available (a) Ground truth(b) SRResNet [19],Trained(c) Bicubic,Not trained(d) deep Prior ,Not trainedFigure 1:Super-resolution using the deep Image method uses a randomly-initialized ConvNet to upsam-ple an Image , using its structure as an Image Prior ; similarto bicubic upsampling, this method does not require learn-ing, but produces much cleaner results with sharper fact, our results are quite close to state-of-the-art super-resolution methods that use ConvNets learned from largedatasets.

3 The deep Image Prior works well for all inverseproblems we could are almost invariably trained on large datasets of im-ages. One may thus assume that their excellent performanceis due to their ability to learn realistic Image priors fromdata. However, learning alone is insufficient to explain thegood performance of deep networks. For instance, the au-thors of [33] recently showed that the same Image classifica-tion network that generalizes well when trained on genuinedata canalsooverfit when presented with random , generalization requires thestructureof the networkto resonate with the structure of the data. However, the9446nature of this interaction remains unclear, particularly in thecontext of Image this work, we show that, contrary to the belief thatlearning is necessary for building good Image priors, a greatdeal of Image statistics are captured by thestructureofa convolutional Image generator independent of is particularly true for the statistics required to solvevarious Image restoration problems, where the Image prioris required to integrate information lost in the show this, we applyuntrainedConvNets to the so-lution of several such problems.

4 Instead of following thecommon paradigm of training a ConvNet on a large datasetof example images, we fit a generator network to a singledegraded Image . In this scheme, the network weights serveas a parametrization of the restored Image . The weights arerandomly initialized and fitted to maximize their likelihoodgiven a specific degraded Image and a task-dependent ob-servation in a different way, we cast reconstruction as acon-ditionalimage generation problem and show that the onlyinformation required to solve it is contained in the singledegraded input imageandthe handcrafted structure of thenetwork used for show that this very simple formulation is very com-petitive for standard Image processing problems such as de-noising, inpainting and super-resolution. This is particu-larly remarkable becauseno aspect of the network is learnedfrom data; instead, the weights of the network are alwaysrandomly initialized, so that the only Prior information is inthe structure of the network itself.

5 To the best of our knowl-edge, this is the first study that directly investigates the priorcaptured by deep convolutional generative networks inde-pendently of learning the network parameters from addition to standard Image restoration tasks, we showan application of our technique to understanding the infor-mation contained within the activations of deep neural net-works. For this, we consider the natural pre- Image tech-nique of [21], whose goal is to characterize the invariantslearned by a deep network by inverting it on the set of nat-ural images. We show that an untrained deep convolutionalgenerator can be used to replace the surrogate natural priorused in [21] (the TV norm) with dramatically improved re-sults. Since the new regularizer, like the TV norm, is notlearned from data but is entirely handcrafted, the resultingvisualizations avoid potential biases arising form the use ofpowerful learned regularizers [8].

6 2. MethodDeep networks are applied to Image generation by learn-ing generator/decoder networksx=f (z)that map a ran-dom code vectorzto an imagex. This approach can be usedto sample realistic images from a random distribution [11].101102103104 Iteration (log scale) + noiseImage shuffledU(0, 1) noiseFigure 2: Learning curves for the reconstruction task us-ing: a natural Image , the same plus noise, the samerandomly scrambled, and white noise. Naturally-lookingimages result in much faster convergence, whereas noise we focus on the case where the distribution iscondi-tionedon a corrupted observationx0to solve inverse prob-lems such as denoising [5] and super-resolution [7].Our aim is to investigate the Prior implicitly captured bythe choice of a particular generator network structure,be-foreany of its parameters are learned. We do so by inter-preting the neural network as aparametrizationx=f (z)of an imagex R3 H W.

7 Herez RC H W is acode tensor/vector and are the network parameters. Thenetwork itself alternates filtering operations such as convo-lution, upsampling and non-linear activation. In particu-lar, most of our experiments are performed using a U-Nettype hourglass architecture with skip-connections, wherezandxhave the same spatial size. Our default architecturehas two million parameters (see Supplementary Materialfor the details of all used architectures).To demonstrate the power of this parametrization, weconsider inverse tasks such as denoising, super-resolutionand inpainting. These can be expressed as energy minimiza-tion problems of the typex = minxE(x;x0) +R(x),(1)whereE(x;x0)is a task-dependent data term,x0thenoisy/low-resolution/occluded Image , andR(x)a choice of data termE(x;x0)is dictated by the appli-cation and will be discussed later. The choice of regularizer,which usually captures a generic Prior on natural images, ismore difficult and is the subject of much research.

8 As asimple example,R(x)can be the Total Variation (TV) ofthe Image , which encourages solutions to contain uniformregions. In this work, wereplacethe regularizerR(x)withthe implicit Prior captured by the neural network, as fol-lows: = argmin E(f (z);x0),x =f (z).(2)9447 The minimizer is obtained using an optimizer such asgradient descent starting from arandom initializationof theparameters. Given a (local) minimizer , the result of therestoration process is obtained asx =f (z). Note thatwhile it is also possible to optimize over the codez, in ourexperiments we do not do that. Thus, unless noted other-wise,zis a fixed3 Dtensor with32feature maps and of thesame spatial size asxfilled with uniform noise. We foundthat additionally perturbingzrandomly at every iterationlead to better results in some experiments ( Supplemen-tary material).In terms of (1), the priorR(x)defined by (2) is an in-dicator functionR(x) = 0for all images that can be pro-duced fromzby a deep ConvNet of a certain architecture,andR(x) = + for all other signals.

9 Since no aspect ofthe network is pre-trained from data, suchdeep Image prioris effectively handcrafted, just like the TV norm. We showthat this hand-crafted Prior works very well for various im-age restoration parametrization with high noise maywonder why a high-capacity networkf can be used as aprior at all. In fact, one may expect to be able to find param-eters recovering any possible imagex, including randomnoise, so that the network should not impose any restrictionon the generated Image . We now show that, while indeedalmost any Image can be fitted, the choice of network ar-chitecture has a major effect on how the solution space issearched by methods such as gradient descent. In partic-ular, we show that the network resists bad solutions anddescends much more quickly towards naturally-looking im-ages. The result is that minimizing (2) either results in agood-looking local optimum, or, at least, the optimizationtrajectory passes near order to study this effect quantitatively, we considerthe most basic reconstruction problem: given a target im-agex0, we want to find the value of the parameters thatreproduce that Image .

10 This can be setup as the optimizationof (2) using a data term comparing the generated Image tox0:E(x;x0) =kx x0k2(3)Plugging this in eq. (2) leads us to the optimization problemmin kf (z) x0k2(4)Figure2shows the value of the energyE(x;x0)as afunction of the gradient descent iterations for four differentchoices for the imagex0: 1) a natural Image , 2) the sameimage plus additive noise, 3) the same Image after randomlypermuting the pixels, and 4) white noise. It is apparent fromthe figure that optimization is much faster for cases 1) and2), whereas the parametrization presents significant iner-tia for cases 3) and 4).Thus, although in the limit the parametrizationcanfit un-structured noise, it does so very reluctantly. In other words,the parametrization offers high impedance to noise and lowimpedance to signal. Therefore for most applications, werestrict the number of iterations in the optimization process(2) to a certain number of iterations.