Noise2Noise: Learning Image Restoration without Clean Data

1 Noise2 Noise: Learning Image Restoration without Clean DataJaakko Lehtinen1 2 Jacob Munkberg1 Jon Hasselgren1 Samuli Laine1 Tero Karras1 Miika Aittala3 Timo Aila1 AbstractWe apply basic statistical reasoning to signal re-construction by machine Learning Learning tomap corrupted observations to Clean signals witha simple and powerful conclusion: it is possi-ble to learn to restore images by only looking atcorrupted examples, at performance at and some-times exceeding training using Clean data, withoutexplicit Image priors or likelihood models of thecorruption. In practice, we show that a singlemodel learns photographic noise removal, denois-ing synthetic Monte Carlo images, and reconstruc-tion of undersampled MRI scans all corruptedby different processes based on noisy data IntroductionSignal reconstruction from corrupted or incomplete mea-surements is an important subfield of statistical data advances in deep neural networks have sparked sig-nificant interest in avoiding the traditional, explicit a prioristatistical modeling of signal corruptions, and insteadlearn-ingto map corrupted observations to the unobserved cleanversions.

2 This happens by training a regression model, ,a convolutional neural network (CNN), with a large numberof pairs( xi,yi)of corrupted inputs xiand Clean targetsyiand minimizing the empirical riskargmin iL(f ( xi), yi),(1)wheref is a parametric family of mappings ( , CNNs),under the loss functionL. We use the notation xto un-derline the fact that the corrupted input x p( x|yi)is arandom variable distributed according to the Clean data may include, for example, pairs of short andlong exposure photographs of the same scene, incompleteand complete k-space samplings of magnetic resonanceimages, fast-but-noisy and slow-but-converged ray-traced1 NVIDIA2 Aalto University3 MIT CSAIL. Correspondence to:Jaakko of the35thInternational Conference on MachineLearning, Stockholm, Sweden, PMLR 80, 2018.

3 Copyright 2018by the author(s).renderings of a synthetic scene, etc. Significant advanceshave been reported in several applications, including Gaus-sian denoising, de-JPEG, text removal (Mao et al., 2016),super-resolution (Ledig et al., 2017), colorization (Zhanget al., 2016), and Image inpainting (Iizuka et al., 2017). Yet,obtaining Clean training targets is often difficult or tedious:a noise-free photograph requires a long exposure; full MRIsampling precludes dynamic subjects; this work, we observe that we can oftenlearn to turnbad images into good images by only looking at bad images,and do this just as well sometimes even better as if wewere using Clean examples. Further, we require neither anexplicit statistical likelihood model of the corruption noran Image prior, and instead learn these indirectly from thetraining data.

4 (Indeed, in one of our examples, syntheticMonte Carlo renderings, the non-stationary noise cannotbe characterized analytically.) In addition to denoising, ourobservation is directly applicable to inverse problems suchas MRI reconstruction from undersampled data. While ourconclusion is almost trivial from a statistical perspective, itsignificantly eases practical learned signal reconstruction bylifting requirements on availability of training reference TensorFlow implementation for Noise2 Noisetraining is available on Theoretical BackgroundAssume that we have a set of unreliable measurements(y1,y2,..)of the room temperature. A common strategyfor estimating the true unknown temperature is to find anumberzthat has the smallest average deviation from themeasurements according to some loss functionL:argminzEy{L(z, y)}.

5 (2)For theL2lossL(z,y) = (z y)2, this minimum is foundat the arithmetic mean of the observations:z=Ey{y}.(3)TheL1loss, the sum of absolute deviationsL(z,y) =|z y|, in turn, has its optimum at the median of the general class of deviation-minimizing estimators are1 [ ] 29 Oct 2018 Noise2 Noise: Learning Image Restoration without Clean Dataknown as M-estimators (Huber, 1964). From a statisticalviewpoint, summary estimation using these common lossfunctions can be seen as ML estimation by interpreting theloss function as the negative log neural network regressors is a generalization ofthis point estimation procedure. Observe the form of thetypical training task for a set of input-target pairs(xi,yi),where the network functionf (x)is parameterized by :argmin E(x,y){L(f (x), y)}.

6 (4)Indeed, if we remove the dependency on input data, anduse a trivialf that merely outputs a learned scalar, the taskreduces to(2). Conversely, the full training task decomposesto the same minimization problem at every training sample;simple manipulations show that (4) is equivalent toargmin Ex{Ey|x{L(f (x), y)}}.(5)The network can, in theory, minimize this loss by solving thepoint estimation problem separately for each input , the properties of the underlying loss are inherited byneural network usual process of training regressors by Equation 1 overa finite number of input-target pairs(xi,yi)hides a subtlepoint: instead of the 1:1 mapping between inputs and tar-gets (falsely) implied by that process, in reality the mappingis multiple-valued.

7 For example, in a superresolution task(Ledig et al., 2017) over all natural images, a low-resolutionimagexcan be explained by many different high-resolutionimagesy, as knowledge about the exact positions and ori-entations of the edges and texture is lost in decimation. Inother words,p(y|x)is the highly complex distribution ofnatural images consistent with the low-resolutionx. Train-ing a neural network regressor using training pairs of low-and high-resolution images using theL2loss, the networklearns to output the average of all plausible explanations( , edges shifted by different amounts), which results inspatial blurriness for the network s predictions. A signif-icant amount of work has been done to combat this wellknown tendency, for example by using learned discriminatorfunctions as losses (Ledig et al.)

8 , 2017; Isola et al., 2017).Our observation is that for certain problems this tendencyhas an unexpected benefit. A trivial, and, at first sight, use-less, property ofL2minimization is that on expectation, theestimate remains unchanged if we replace the targets withrandom numbers whose expectations match the targets. Thisis easy to see: Equation(3)holds, no matter what particu-lar distribution theys are drawn from. Consequently, theoptimal network parameters of Equation(5)also remainunchanged, if input-conditioned target distributionsp(y|x)are replaced with arbitrary distributions that have the sameconditional expected implies that we can, inprinciple, corrupt the training targets of a neural networkwith zero-mean noise without changing what the this with the corrupted inputs from Equa-tion 1, we are left with the empirical risk minimization taskargmin iL(f ( xi), yi),(6)where both the inputs and the targets are now drawn froma corrupted distribution (not necessarily the same), condi-tioned on the underlying, unobserved Clean targetyisuchthatE{ yi| xi}=yi.

9 Given infinite data, the solution isthe same as that of(1). For finite data, the variance is theaverage variance of the corruptions in the targets, dividedby the number of training samples (see appendix). Inter-estingly, none of the above relies on a likelihood model ofthe corruption, nor a density model (prior) for the under-lying Clean Image manifold. That is, we do not need anexplicitp(noisy| Clean )orp( Clean ), as long as we have datadistributed according to many Image Restoration tasks, the expectation of the cor-rupted input dataisthe Clean target that we seek to photography is an example: a long, noise-free ex-posure is the average of short, independent, noisy this in mind, the above suggests the ability to learn toremove photon noise given only pairs of noisy images, withno need for potentially expensive or difficult long observations can be made about other loss instance, theL1loss recovers the median of the targets,meaning that neural networks can be trained to repair im-ages with significant (up top 50%) outlier content, againonly requiring access to pairs of such corrupted the next sections, we present a wide variety of examplesdemonstrating that these theoretical capabilities are alsoefficiently realizable in Practical ExperimentsWe now experimentally study the practical properties ofnoisy-target training.

10 We start with simple noise distribu-tions (Gaussian, Poisson, Bernoulli) in Sections and ,and continue to the much harder, analytically intractableMonte Carlo Image synthesis noise (Section ). In Sec-tion , we show that Image reconstruction from sub-Nyquist spectral samplings in magnetic resonance imaging(MRI) can be learned from corrupted observations Additive Gaussian NoiseWe will first study the effect of corrupted targets usingsynthetic additive Gaussian noise. As the noise has zeromean, we use theL2loss for training to recover the baseline is a recent state-of-the-art method RED30 (Mao et al., 2016), a 30-layer hierarchical residual net-Noise2 Noise: Learning Image Restoration without Clean targetsnoisy pix5 pix10 pix20 pix40 1 (trad.)