Rectified Linear Units Improve Restricted Boltzmann Machines

1 Rectified Linear Units Improve Restricted Boltzmann MachinesVinod E. of Computer Science, University of Toronto, Toronto, ON M5S 2G4, CanadaAbstractRestricted Boltzmann Machines were devel-oped using binary stochastic hidden can be generalized by replacing eachbinary unit by an infinite number of copiesthat all have the same weights but have pro-gressively more negative biases. The learningand inference rules for these Stepped Sig-moid Units are unchanged. They can be ap-proximated efficiently by noisy, Rectified lin-ear Units . Compared with binary Units , theseunits learn features that are better for objectrecognition on the NORB dataset and faceverification on the Labeled Faces in the Wilddataset.

2 Unlike binary Units , Rectified linearunits preserve information about relative in-tensities as information travels through mul-tiple layers of feature IntroductionRestricted Boltzmann Machines (RBMs) have beenused as generative models of many different typesof data including labeled or unlabeled images(Hinton et al.,2006), sequences of mel-cepstral coef-ficients that represent speech (Mohamed & Hinton,2010), bags of words that represent documents(Salakhutdinov & Hinton,2009), and user ratings ofmovies (Salakhutdinov et al.,2007).

3 In their con-ditional form they can be used to model high-dimensional temporal sequences such as video or mo-tion capture data (Taylor et al.,2006). Their most im-portant use is as learning modules that are composedto form deep belief nets (Hinton et al.,2006).Appearing inProceedings of the 27thInternational Confer-ence on Machine Learning, Haifa, Israel, 2010. Copyright2010 by the author(s)/owner(s). Learning a Restricted Boltzmann MachineImages composed of binary pixels can be modeled byan RBM that uses a layer of binary hidden Units (fea-ture detectors) to model the higher-order correlationsbetween pixels.

4 If there are no direct interactions be-tween the hidden Units and no direct interactions be-tween the visible Units that represent the pixels, thereis a simple and efficient method called ContrastiveDivergence to learn a good set of feature detectorsfrom a set of training images (Hinton,2002). We startwith small, random weights on the symmetric connec-tions between each pixeliand each feature we repeatedly update each weight,wij, using thedifference between two measured, pairwise correlations wij= (<vihj>data <vihj>recon)(1)where is a learning rate,<vihj>datais the frequencywith which visible unitiand hidden unitjare on to-gether when the feature detectors are being driven byimages from the training set and< vihj>reconis thecorresponding frequency when the hidden Units are be-ing driven by reconstructed images.

5 A similar learningrule can be used for the a training image, we set the binary state,hj, ofeach feature detector to be 1 with probabilityp(hj= 1) =11 + exp( bj i visviwij)(2)wherebjis the bias ofjandviis the binary stateof pixeli. Once binary states have been chosen forthe hidden Units we produce a reconstruction of thetraining image by setting the state of each pixel to be1 with probabilityp(vi= 1) =11 + exp( bi j hidhjwij)(3)The learned weights and biases implicitly define aprobability distribution over all possible binary imagesvia the energy,E(v,h), of a joint configuration of theReLUs Improve RBMsvisible and hidden Units .

6 E(v,h) = i,jvihjwij ivibi jhjbj(4)p(v) = he E(v,h) u,ge E(u,g)(5) Gaussian unitsRBMs were originally developed using binary stochas-tic Units for both the visible and hidden layers(Hinton,2002). To deal with real-valued datasuch as the pixel intensities in natural images,(Hinton & Salakhutdinov,2006) replaced the binaryvisible Units by Linear Units with independent Gaus-sian noise as first suggested by (Freund & Haussler,1994). The energy function then becomes:E(v,h) = i vis(vi bi)22 2i j hidbjhj i,jvi ihjwij(6)where iis the standard deviation of the Gaussiannoise for visible is possible to learn the variance of the noise for eachvisible unit but this is difficult using binary hiddenunits.

7 In many applications, it is much easier to firstnormalise each component of the data to have zeromean and unit variance and then to use noise-free re-constructions, with the variance in equation6set to reconstructed value of a Gaussian visible unit isthen equal to its top-down input from the binary hid-den Units plus its bias. We use this type of noise-freevisible unit for the models of object and face imagesdescribed Rectified Linear unitsTo allow each unit to express more information,(Teh & Hinton,2001) introduced binomial Units whichcan be viewed asNseparate copies of a binary unitthat all share the same bias and weights.

8 A nice side-effect of using weight-sharing to synthesize a new typeof unit out of binary Units is that the mathematicsunderlying learning in binary-binary RBM s remainsunchanged. Since allNcopies receive the same totalinput, they all have the same probability,p, of turn-ing on and this only has to be computed once. Theexpected number that are on isN pand the variancein this number isN p(1 p). For smallp, this acts likea Poisson unit , but aspapproaches 1 the variance be-comes small again which may not be desireable. Also,for small values ofpthe growth inpis exponential inthe total input.

9 This makes learning much less stable 505100246810 Figure comparison of three different ways to modelrectified Linear Units . The red curve shows the expectedvalue of the sum of an infinite number of binary Units witheach having a bias one less than the previous one. The bluecurve is the approximation log(1+exp(x)). The green curveis the expected value of a Rectified Linear unit with addedGaussian noise as described in section2. The red and bluecurves are virtually for the stepped sigmoid Units (SSU) small modification to binomial Units makes themfar more interesting as models of real neurons and alsomore useful for practical applications.

10 We make an in-finite number of copies that all have the same learnedweight vectorwand the same learned bias,b, but eachcopy has a different, fixed offset to the bias. If the off-sets are , , , ..the sum of the probabili-ties of the copies is extremely close to having a closedform (figure1):N i=1 (x i+ ) log(1 +ex),(7)wherex=vwT+ the total activity of all of the copies behaves likea noisy, integer-valued version of a smoothed rectifiedlinear unit1. Even though log(1 +ex) is not in theexponential family, we can model it accurately usinga set of binary Units with shared weights and fixedbias offsets.