A Fast Learning Algorithm for Deep Belief Nets

LETTERC ommunicated by Yann Le CunA fast Learning Algorithm for deep Belief NetsGeoffrey E. of Computer Science, University of Toronto, Toronto, Canada M5S 3G4 Yee-Whye of Computer Science, National University of Singapore,Singapore 117543We show how to use complementary priors to eliminate the explaining-away effects that make inference difficult in densely connected Belief netsthat have many hidden layers. Using complementary priors, we derive afast, greedy Algorithm that can learn deep , directed Belief networks onelayer at a time, provided the top two layers form an undirected associa-tive memory. The fast , greedy Algorithm is used to initialize a slowerlearning procedure that fine-tunes the weights using a contrastive ver-sion of the wake-sleep Algorithm . After fine-tuning, a network with threehidden layers forms a very good generative model of the joint distribu-tion of handwritten digit images and their labels.

A Fast Learning Algorithm for Deep Belief Nets 1531 weights, w ij, on the directed connections from the ancestors: p(s i = 1) = 1 1 +exp −b i − j s jw ij, (2.1) where b i is the bias of unit i.If a logistic belief net has only one hidden layer, the prior distribution over the hidden variables is factorial because

Loading..

Tags:

  Learning, Deep, Fast, Algorithm, Belief, Fast learning algorithm for deep belief

Information

Related search queries