Transcription of Deep Sparse Recti er Neural Networks
{{id}} {{{paragraph}}}
315 Deep Sparse Rectifier Neural NetworksXavier GlorotAntoine BordesYoshua BengioDIRO, Universit e de Montr ealMontr eal, QC, UMR CNRS 6599 UTC, Compi`egne, FranceandDIRO, Universit e de Montr ealMontr eal, QC, Universit e de Montr ealMontr eal, QC, logistic sigmoid neurons are more bi-ologically plausible than hyperbolic tangentneurons, the latter work better for train-ing multi-layer Neural Networks . This pa-per shows that rectifying neurons are aneven better model of biological neurons andyield equal or better performance than hy-perbolic tangent Networks in spite of thehard non-linearity and non-differentiabilityat zero, creating Sparse representations withtrue zeros, which seem remarkably suitablefor naturally Sparse data.
tion regime. This is biologically implausible and hurts gradient-based optimization (LeCun et al., 1998; Bengio and Glorot, 2010). Important divergences between biological and machine learning models concern non-linear activation functions. A common biological model of neuron, the leaky integrate-and- re (or LIF) (Dayan and Abott, 2001), gives ...
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document:
{{id}} {{{paragraph}}}