Extracting and Composing Robust Features with Denoising ...

1 Extracting and Composing Robust Features withDenoising AutoencodersPascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine ManzagolDept. IRO, Universit e de Montr 6128, Montreal, Qc, H3C 3J7, lisaTechnical Report 1316, February 2008 AbstractPrevious work has shown that the difficulties in learning deep genera-tive or discriminative models can be overcome by an initial unsupervisedlearning step that maps inputs to useful intermediate representations. Weintroduce and motivate a new training principle for unsupervised learningof a representation based on the idea of making the learned representa-tions Robust to partial corruption of the input pattern. This approach canbe used to train autoencoders, and these Denoising autoencoders can bestacked to initialize deep architectures. The algorithm can be motivatedfrom a manifold learning and information theoretic perspective or from agenerative model perspective.

2 Comparative experiments clearly show thesurprising advantage of corrupting the input of autoencoders on a patternclassification benchmark IntroductionRecent theoretical studies indicate that deep architectures (Bengio & Le Cun,2007; Bengio, 2007) may be needed toefficientlymodel complex distributionsand achieve better generalization performance on challenging recognition belief that additional levels of functional composition will yield increasedrepresentational and modeling power is not new (McClelland et al., 1986; Hin-ton, 1989; Utgoff & Stracuzzi, 2002). However, in practice, learning in deeparchitectures has proven to be difficult. One needs only to ponder the diffi-cult problem of inference in deep directed graphical models, due to explainingaway . Also looking back at the history of multi-layer neural networks, theirdifficult optimization (Bengio et al., 2007; Bengio, 2007) has long preventedreaping the expected benefits of going beyond one or two hidden layers.

3 How-ever this situation has recently changed with the successful approach of (Hintonet al., 2006; Hinton & Salakhutdinov, 2006; Bengio et al., 2007; Ranzato et al.,2007; Lee et al., 2008) for training Deep Belief Networks and stacked key ingredient to this success appears to be the use of an unsupervisedtraining criterion to perform a layer-by-layer initialization: each layer is at firsttrained to produce a higher level (hidden) representation of the observed pat-terns, based on the representation it receives as input from the layer below, byoptimizing a local unsupervised criterion. Each level produces a representationof the input pattern that is more abstract than the previous level s, because itis obtained by Composing more operations. This initialization yields a startingpoint, from which a global fine-tuning of the model s parameters is then per-formed using another training criterion appropriate for the task at hand.

4 Thistechnique has been shown empirically to avoid getting stuck in the kind of poorsolutions one typically reaches with random initializations. While unsupervisedlearning of a mapping that produces good intermediate representations ofthe input pattern seems to be key, little is understood regarding what consti-tutes good representations for initializing deep architectures, or what explicitcriteria may guide learning such representations. We know of only a few algo-rithms that seem to work well for this purpose: Restricted Boltzmann Machines(RBMs) trained with contrastive divergence on one hand, and various types ofautoencoders on the present research begins with the question of what explicit criteria a goodintermediate representation should satisfy. Obviously, it should at a minimumretain a certain amount of information about its input, while at the same timebeing constrained to a given form ( a real-valued vector of a given size in thecase of an autoencoder).

5 A supplemental criterion that has been proposed forsuch models is sparsity of the representation (Ranzato et al., 2008; Lee et al.,2008). Here we hypothesize and investigate an additional specific criterion:robustness to partial destruction of the input, , partially destructedinputs should yield almost the same representation. It is motivated by thefollowing informal reasoning: a good representation is expected to capture stablestructures in the form of dependencies and regularities characteristic of the(unknown) distribution of its observed input. For high dimensional redundantinput (such as images) at least, such structures are likely to depend on evidencegathered from a combination of many input dimensions. They should thus berecoverable from partial observation only. A hallmark of this is our humanability to recognize partially occluded or corrupted images. Further evidence isour ability to form a high level concept associated to multiple modalities (suchas image and sound) and recall it even when some of the modalities are validate our hypothesis and assess its usefulness as one of the guidingprinciples in learning deep architectures, we propose a modification to the au-toencoder framework to explicitly integrate robustness to partially destroyedinputs.

6 Section 2 describes the algorithm in details. Section 3 discusses linkswith other approaches in the literature. Section 4 is devoted to a closer inspec-tion of the model from different theoretical standpoints. In section 5 we verifyempirically if the algorithm leads to a difference in performance. Section 6concludes the Description of the Notation and SetupLetXandYbe two random variables with joint probability densityp(X,Y),with marginal distributionsp(X) andp(Y). Throughout the text, we willuse the following notation: Expectation:EEp(X)[f(X)] = p(x)f(x)dx. En-tropy:IH(X) =IH(p) =EEp(X)[ logp(X)]. Conditional entropy:IH(X|Y) =EEp(X,Y)[ logp(X|Y)]. Kullback-Leibler divergence:IDKL(p q) =EEp(X)[logp(X)q(X)].Cross-entropy:IH(p q) =EEp(X)[ logq(X)] =IH(p) +IDKL(p q). Mutual infor-mation:I(X;Y) =IH(X) IH(X|Y). Sigmoid:s(x) =11+e xands(x) =(s(x1),..,s(xd))T. Bernoulli distribution with mean :B (x). and by exten-sionB (x) = (B 1(x1).)

7 ,B d(xd)).The setup we consider is the typical supervised learning setup with a trainingset ofn(input, target) pairsDn={(x(1),t(1))..,(x(n),t(n))}, that we supposeto be an sample from an unknown distributionq(X,T) with correspondingmarginalsq(X) andq(T). The Basic AutoencoderWe begin by recalling the traditional autoencoder model such as the one usedin (Bengio et al., 2007) to build deep networks. An autoencoder takes an inputvectorx [0,1]d, and first maps it to a hidden representationy [0,1]d through a deterministic mappingy=f (x) =s(Wx+b), parameterized by ={W,b}.Wis ad dweight matrix andbis a bias vector. The resultinglatent representationyis then mapped back to a reconstructed vectorz [0,1]din input spacez=g (y) =s(W y+b ) with ={W ,b }. The weightmatrixW of the reverse mapping may optionally be constrained byW =WT,in which case the autoencoder is said to havetied weights. Each trainingx(i)isthus mapped to a correspondingy(i)and a reconstructionz(i).

8 The parametersof this model are optimized to minimize theaverage reconstruction error: ?, ?=arg min , 1nn i=1L(x(i),z(i))=arg min , 1nn i=1L(x(i),g (f (x(i))))(1)whereLis a loss function such as the traditionalsquared errorL(x,z) = x z alternative loss, suggested by the interpretation ofxandzas either bitvectors or vectors of bit probabilities (Bernoullis) is thereconstruction cross-entropy:LIH(x,z)=IH(Bx Bz)= d k=1[xklogzk+(1 xk) log(1 zk)](2)3 Figure 1: An examplexis corrupted to x. The autoencoder then maps it toyand attempts to xxyg Note that ifxis a binary vector,LIH(x,z) is a negative log-likelihood for theexamplex, given the Bernoulli parametersz. Equation 1 withL=LIHcan bewritten ?, ?= arg min , EEq0(X)[LIH(X,g (f (X)))](3)whereq0(X) denotes the empirical distribution associated to ourntraininginputs. This optimization will typically be carried out by stochastic The Denoising AutoencoderTo test our hypothesis and enforce robustness to partially destroyed inputs wemodify the basic autoencoder we just described.

9 We will now train it to recon-struct a clean repaired input from acorrupted, partially destroyed one. Thisis done by first corrupting the initial inputxto get a partially destroyed ver-sion xby means of a stochastic mapping x qD( x|x). In our experiments, weconsidered the following corrupting process, parameterized by the desired pro-portion of destruction : for each inputx, a fixed number dof componentsare chosen at random, and their value is forced to 0, while the others are leftuntouched. The procedure can be viewed as replacing a component consideredmissing by a default value, which is a common technique. A motivation forzeroing the destroyed components is that it simulates the removal of these com-ponents from the input. For images on a white (0) background, this correspondsto salt noise . Note that alternative corrupting noises could be corrupted input xis then mapped, as with the basic autoencoder, to ahidden representationy=f ( x) =s(W x+b) from which we reconstruct az=g (y) =s(W y+b ) (see figure 1 for a schematic representation of theprocess).

10 As before the parameters are trained to minimize the average recon-struction errorLIH(x,z) =IH(Bx Bz) over a training set, to havezas closeas possible to the uncorrupted inputx. But the key difference is thatzis nowa deterministic function of xrather thanxand thus the result of a stochasticmapping us define the joint distributionq0(X, X,Y) =q0(X)qD( X|X) f (eX)(Y)(4)1the approach we describe and our analysis is not specific to a particular kind of u(v) puts mass 0 whenu6=v. ThusYis a deterministic function of (X, X,Y) is parameterized by . The objective function minimized bystochastic gradient descent becomes:arg min , EEq0(X,eX)[LIH(X,g (f ( X)))].(5)So from the point of view of the stochastic gradient descent algorithm, in ad-dition to picking an input sample from the training set, we will also produce arandom corrupted version of it, and take a gradient step towards reconstructingthe uncorrupted version from the corrupted version.