Rectified Linear Units Improve Restricted Boltzmann Machines

1 Rectified Linear Units Improve Restricted Boltzmann MachinesVinod E. of Computer Science, University of Toronto, Toronto, ON M5S 2G4, CanadaAbstractRestricted Boltzmann Machines were devel-oped using binary stochastic hidden can be generalized by replacing eachbinary unit by an infinite number of copiesthat all have the same weights but have pro-gressively more negative biases. The learningand inference rules for these Stepped Sig-moid Units are unchanged. They can be ap-proximated efficiently by noisy, Rectified lin-ear Units . Compared with binary Units , theseunits learn features that are better for objectrecognition on the NORB dataset and faceverification on the Labeled Faces in the Wilddataset. Unlike binary Units , Rectified linearunits preserve information about relative in-tensities as information travels through mul-tiple layers of feature IntroductionRestricted Boltzmann Machines (RBMs) have beenused as generative models of many different typesof data including labeled or unlabeled images(Hinton et al.)

2 ,2006), sequences of mel-cepstral coef-ficients that represent speech (Mohamed & Hinton,2010), bags of words that represent documents(Salakhutdinov & Hinton,2009), and user ratings ofmovies (Salakhutdinov et al.,2007). In their con-ditional form they can be used to model high-dimensional temporal sequences such as video or mo-tion capture data (Taylor et al.,2006). Their most im-portant use is as learning modules that are composedto form deep belief nets (Hinton et al.,2006).Appearing inProceedings of the 27thInternational Confer-ence on Machine Learning, Haifa, Israel, 2010. Copyright2010 by the author(s)/owner(s). Learning a Restricted Boltzmann MachineImages composed of binary pixels can be modeled byan RBM that uses a layer of binary hidden Units (fea-ture detectors) to model the higher-order correlationsbetween pixels. If there are no direct interactions be-tween the hidden Units and no direct interactions be-tween the visible Units that represent the pixels, thereis a simple and efficient method called ContrastiveDivergence to learn a good set of feature detectorsfrom a set of training images (Hinton,2002).

3 We startwith small, random weights on the symmetric connec-tions between each pixeliand each feature we repeatedly update each weight,wij, using thedifference between two measured, pairwise correlations wij= (<vihj>data <vihj>recon)(1)where is a learning rate,<vihj>datais the frequencywith which visible unitiand hidden unitjare on to-gether when the feature detectors are being driven byimages from the training set and< vihj>reconis thecorresponding frequency when the hidden Units are be-ing driven by reconstructed images. A similar learningrule can be used for the a training image, we set the binary state,hj, ofeach feature detector to be 1 with probabilityp(hj= 1) =11 + exp( bj i visviwij)(2)wherebjis the bias ofjandviis the binary stateof pixeli. Once binary states have been chosen forthe hidden Units we produce a reconstruction of thetraining image by setting the state of each pixel to be1 with probabilityp(vi= 1) =11 + exp( bi j hidhjwij)(3)The learned weights and biases implicitly define aprobability distribution over all possible binary imagesvia the energy,E(v,h), of a joint configuration of theReLUs Improve RBMsvisible and hidden Units :E(v,h) = i,jvihjwij ivibi jhjbj(4)p(v) = he E(v,h) u,ge E(u,g)(5) Gaussian unitsRBMs were originally developed using binary stochas-tic Units for both the visible and hidden layers(Hinton,2002).

4 To deal with real-valued datasuch as the pixel intensities in natural images,(Hinton & Salakhutdinov,2006) replaced the binaryvisible Units by Linear Units with independent Gaus-sian noise as first suggested by (Freund & Haussler,1994). The energy function then becomes:E(v,h) = i vis(vi bi)22 2i j hidbjhj i,jvi ihjwij(6)where iis the standard deviation of the Gaussiannoise for visible is possible to learn the variance of the noise for eachvisible unit but this is difficult using binary hiddenunits. In many applications, it is much easier to firstnormalise each component of the data to have zeromean and unit variance and then to use noise-free re-constructions, with the variance in equation6set to reconstructed value of a Gaussian visible unit isthen equal to its top-down input from the binary hid-den Units plus its bias. We use this type of noise-freevisible unit for the models of object and face imagesdescribed Rectified Linear unitsTo allow each unit to express more information,(Teh & Hinton,2001) introduced binomial Units whichcan be viewed asNseparate copies of a binary unitthat all share the same bias and weights.

5 A nice side-effect of using weight-sharing to synthesize a new typeof unit out of binary Units is that the mathematicsunderlying learning in binary-binary RBM s remainsunchanged. Since allNcopies receive the same totalinput, they all have the same probability,p, of turn-ing on and this only has to be computed once. Theexpected number that are on isN pand the variancein this number isN p(1 p). For smallp, this acts likea Poisson unit , but aspapproaches 1 the variance be-comes small again which may not be desireable. Also,for small values ofpthe growth inpis exponential inthe total input. This makes learning much less stable 505100246810 Figure comparison of three different ways to modelrectified Linear Units . The red curve shows the expectedvalue of the sum of an infinite number of binary Units witheach having a bias one less than the previous one. The bluecurve is the approximation log(1+exp(x)). The green curveis the expected value of a Rectified Linear unit with addedGaussian noise as described in section2.

6 The red and bluecurves are virtually for the stepped sigmoid Units (SSU) small modification to binomial Units makes themfar more interesting as models of real neurons and alsomore useful for practical applications. We make an in-finite number of copies that all have the same learnedweight vectorwand the same learned bias,b, but eachcopy has a different, fixed offset to the bias. If the off-sets are , , , ..the sum of the probabili-ties of the copies is extremely close to having a closedform (figure1):N i=1 (x i+ ) log(1 +ex),(7)wherex=vwT+ the total activity of all of the copies behaves likea noisy, integer-valued version of a smoothed rectifiedlinear unit1. Even though log(1 +ex) is not in theexponential family, we can model it accurately usinga set of binary Units with shared weights and fixedbias offsets. This set has no more parameters than anordinary binary unit , but it provides a much more ex-pressive variable. The variance in the integer activitylevel is (x) so Units that are firmly off do not createnoise and the noise does not become large drawback of giving each copy a bias that differsby a fixed offset is that the logistic sigmoid functionneeds to be used many times to get the probabilities1If we only useNcopies, we need to subtract the termlog(1 +ex N) from the Improve RBMsrequired for sampling an integer value correctly.

7 Itis possible, however, to use a fast approximation inwhich the sampled value of the Rectified Linear unit isnot constrained to be an integer. Instead it is given bymax(0, x+N(0, (x)) whereN(0, V) is Gaussian noisewith zero mean and varianceV. We call a unit thatuses this approximation aNoisyRectifiedLinearUnit(NReLU) and this paper shows that NReLUs workbetter than binary hidden Units for several differenttasks.(Jarrett et al.,2009) have explored various rectifiednonlinearities (including themax(0, x) nonlinearity,which they refer to as positive part ) in the con-text of convolutional networks and have found themto Improve discriminative performance. Our empiricalresults in sections5and6further support this ob-servation. We also give an approximate probabilisticinterpretation for themax(0, x) nonlinearity, furtherjustifying their Intensity equivarianceNReLU s have some interesting mathematical proper-ties (Hahnloser et al.,2003), one of which is very use-ful for object recognition.)

8 A major consideration whendesigning an object recognition system is how to makethe output invariant to properties of the input such aslocation, scale, orientation, lightingetc. Convolutionalneural networks are often said to achieve translationinvariance but in their pure form they actually achievesomething quite different. If an object is translated inthe input image, its representation in a pool of localfilters that have shared weights is also translated. Soif it can be represented well by a pattern of featureactivities when it is in one location, it can also be rep-resented equally well by atranslatedpattern of featureactivities when it is another location. We call thistranslation equivariance: the representation varies inthe same way as the image. In a deep convolutionalnet, translation invaraince is achieved by using sub-sampling to introduce a small amount of translationinvariance after each layer of hidden Units do not exhibit intensity equivari-ance, but Rectified Linear Units do, provided they havezero biases and are noise-free.

9 Scaling up all of the in-tensities in an image by >0 cannot change whethera zero-bias unit receives a total input above or belowzero. So all of the off Units remain off and the re-mainder all increase their activities by a factor of .This stays true for many layers of Rectified Linear deciding whether two face images come from thesame person, we make use of this nice property of rec-tified Linear Units by basing the decision on the cosineof the angle between the activities of the feature detec-tors in the last hidden layer. The feature vectors areintensity equivariant and the cosine is intensity invari-ant. The type of intensity invariance that is importantfor recognition cannot be achieved by simply dividingall the pixel intensities by their sum. This would causea big change in the activities of feature detectors thatattend to the parts of a face when there is a brightspot in the Empirical EvaluationWe empirically compare NReLUs to stochastic bi-nary hidden units2on two vision tasks: 1) objectrecognition on the Jittered-Cluttered NORB dataset(LeCun et al.

10 ,2004), and 2) face verification on theLabeled Faces in the Wild dataset (Huang et al.,2007). Both datasets contain complicated image vari-ability that make them difficult tasks. Also, they bothalready have a number of published results for variousmethods, which gives a convenient basis for judginghow good our results are. We use RBMs with binaryhidden Units or NReLUs to generatively pre-train oneor more layers of features and we then discriminativelyfine-tune the features using backpropagation. On bothtasks NReLUs give better discriminative performancethan binary Units . The discriminative models use thedeterministic version of NReLUs that implement thefunctiony=max(0, x). For backpropagation, we takethe gradient of this function to be 0 whenx 0 and 1whenx >0 ( we ignore the discontinuity atx= 0).5. Jittered-Cluttered NORBNORB is a synthetic 3D object recognition datasetthat contains five classes of toys (humans,animals,cars,planes,trucks) imaged by a stereo-pair cam-era system from different viewpoints under differentlighting conditions.