Deep Sparse Recti er Neural Networks

1 315 Deep Sparse Rectifier Neural NetworksXavier GlorotAntoine BordesYoshua BengioDIRO, Universit e de Montr ealMontr eal, QC, UMR CNRS 6599 UTC, Compi`egne, FranceandDIRO, Universit e de Montr ealMontr eal, QC, Universit e de Montr ealMontr eal, QC, logistic sigmoid neurons are more bi-ologically plausible than hyperbolic tangentneurons, the latter work better for train-ing multi-layer Neural Networks . This pa-per shows that rectifying neurons are aneven better model of biological neurons andyield equal or better performance than hy-perbolic tangent Networks in spite of thehard non-linearity and non-differentiabilityat zero, creating Sparse representations withtrue zeros, which seem remarkably suitablefor naturally Sparse data.

2 Even though theycan take advantage of semi-supervised setupswith extra-unlabeled data, deep rectifier net-works can reach their best performance with-out requiring any unsupervised pre-trainingon purely supervised tasks with large labeleddatasets. Hence, these results can be seen asa new milestone in the attempts at under-standing the difficulty in training deep butpurely supervised Neural Networks , and clos-ing the performance gap between Neural net-works learnt with and without IntroductionMany differences exist between the Neural networkmodels used by machine learning researchers and thoseused by computational neuroscientists.

3 This is in partAppearing in Proceedings of the 14thInternational Con-ference on Artificial Intelligence and Statistics (AISTATS)2011, Fort Lauderdale, FL, USA. Volume 15 of JMLR:W&CP 15. Copyright 2011 by the the objective of the former is to obtain com-putationally efficient learners, that generalize well tonew examples, whereas the objective of the latter is toabstract out neuroscientific data while obtaining ex-planations of the principles involved, providing predic-tions and guidance for future biological where both objectives coincide are thereforeparticularly worthy of investigation, pointing towardscomputationally motivated principles of operation inthe brain that can also enhance research in artificialintelligence.

4 In this paper we show that two com-mon gaps between computational neuroscience modelsand machine learning Neural network models can bebridged by using the following linear by part activa-tion : max(0,x), called the rectifier (or hinge) activa-tion function. Experimental results will show engagingtraining behavior of this activation function, especiallyfordeep architectures(see Bengio (2009) for a review), , where the number of hidden layers in the neuralnetwork is 3 or theoretical and empirical work in statisticalmachine learning has demonstrated the importance oflearning algorithms for deep architectures. This is inpart inspired by observations of the mammalian vi-sual cortex, which consists of a chain of processingelements, each of which is associated with a differentrepresentation of the raw visual input.

5 This is partic-ularly clear in the primate visual system (Serreet al.,2007), with its sequence of processing stages: detectionof edges, primitive shapes, and moving up to gradu-ally more complex visual shapes. Interestingly, it wasfound that the features learned in deep architecturesresemble those observed in the first two of these stages(in areas V1 and V2 of visual cortex) (Leeet al., 2008),and that they become increasingly invariant to factorsof variation (such as camera movement) in higher lay-ers (Goodfellowet al., 2009). 316 Deep Sparse Rectifier Neural NetworksRegarding the training of deep Networks , somethingthat can be considered a breakthrough happenedin 2006, with the introduction of Deep Belief Net-works (Hintonet al.)

6 , 2006), and more generally theidea of initializing each layer by unsupervised learn-ing (Bengioet al., 2007; Ranzatoet al., 2007). Someauthors have tried to understand why this unsuper-vised procedure helps (Erhanet al., 2010) while oth-ers investigated why the original training procedure fordeep Neural Networks failed (Bengio and Glorot, 2010).From the machine learning point of view, this paperbrings additional results in these lines of propose to explore the use of rectifying non-linearities as alternatives to the hyperbolic tangentor sigmoid in deep artificial Neural Networks , in ad-dition to using anL1regularizer on the activation val-ues to promote sparsity and prevent potential numer-ical problems with unbounded activation.

7 Nair andHinton (2010) present promising results of the influ-ence of such units in the context of Restricted Boltz-mann Machines compared to logistic sigmoid activa-tions on image classification tasks. Our work extendsthis for the case of pre-training using denoising auto-encoders (Vincentet al., 2008) and provides an exten-sive empirical comparison of the rectifying activationfunction against the hyperbolic tangent on image clas-sification benchmarks as well as an original derivationfor the text application of sentiment experiments on image and text data indicate thattraining proceeds better when the artificial neurons areeither off or operating mostly in a linear regime.

8 Sur-prisingly, rectifying activation allows deep Networks toachieve their best performance without unsupervisedpre-training. Hence, our work proposes a new contri-bution to the trend of understanding and merging theperformance gap between deep Networks learnt withand without unsupervised pre-training (Erhanet al.,2010; Bengio and Glorot, 2010). Still, rectifier net-works can benefit from unsupervised pre-training inthe context of semi-supervised learning where largeamounts of unlabeled data are provided. Furthermore,as rectifier units naturally lead to Sparse Networks andare closer to biological neurons responses in their mainoperating regime, this work also bridges (in part) amachine learning / neuroscience gap in terms of acti-vation function and paper is organized as follows.

9 Section 2 presentssome neuroscience and machine learning backgroundwhich inspired this work. Section 3 introduces Recti -fier neurons and explains their potential benefits anddrawbacks in deep Networks . Then we propose anexperimental study with empirical results on imagerecognition in Section and sentiment analysis inSection Section 5 presents our Neuroscience ObservationsFor models of biological neurons, the activation func-tion is the expected firing rate as a function of thetotal input currently arising out of incoming signalsat synapses (Dayan and Abott, 2001). An activationfunction is termed, respectivelyantisymmetricorsym-metricwhe n its response to the opposite of a stronglyexcitatory input pattern is respectively a strongly in-hibitory or excitatory one, andone-sidedwhen thisresponse is zero.

10 The main gaps that we wish to con-sider between computational neuroscience models andmachine learning models are the following: Studies on brain energy expense suggest thatneurons encode information in a Sparse and dis-tributed way (Attwell and Laughlin, 2001), esti-mating the percentage of neurons active at thesame time to be between 1 and 4% (Lennie, 2003).This corresponds to a trade-off between richnessof representation and small action potential en-ergy expenditure. Without additional regulariza-tion, such as anL1penalty, ordinary feedforwardneural nets do not have this property. For ex-ample, the sigmoid activation has a steady stateregime around12, therefore, after initializing withsmall weights, all neurons fire at half their satura-tion regime.