ディープラーニング - Image Analysis Lab

1 deep Learning deep Learning = Shallow NN deep Neural Network (DNN) .. DNN NN NN Encoder-decoder paradigm Convolutional Neural Networks (CNN) CNN vs. fully-connected NN Restricted Boltzmann Machine (RBM) deep Belief Network (DBN) cuda-convnet MNIST DNN NN NN Encoder-decoder paradigm Convolutional Neural Networks (CNN) CNN vs.

2 Fully-connected NN Restricted Boltzmann Machine (RBM) deep Belief Network (DBN) cuda-convnet MNIST 12 NN 10 16 PC1000 PC 3 YouTube 1000 The New York Times (2012/6/25) Ng Le et al., Building High-level Features Using Large Scale Unsupervised Learning , ICML2012 : Hinton IMAGENET Large Scale Visual Recognition Challenge 2012 1000 1000 accordion.

3 Airliner Team name Error (5 guesses) 1 SuperVision 2 ISI 3 OXFORD_VGG 4 XRCE/INRIA 5 University of Amsterdam 6 LEAR-XRCE 7-layer NN Krizhevsk et al., ImageNet Classification with deep Convolutional Neural Networks, NIPS2012 Schmidhuber IDSIA 1 IJCNN 2011 Traffic Sign Recognition Competition; 1st ( ), 2nd ( , Humans), 3rd ( ), 4th ( ) ICPR 2012 Contest on Mitosis Detection in Breast Cancer Histological Images ISBI 2012 challenge on segmentation of neuronal structures ICDAR 2011 Offline Chinese Handwriting Competition ICDAR2009 Arabic Connected Handwriting Competition French Connected Handwriting Competition (2012 11 ) NORB object recognition benchmark CIFAR Image classification benchmark MNIST handwritten digits benchmark; human-competitive result Ciresan et al.

4 , Multi-column deep Neural Networks for Image Classification, CVPR2012 speech recognition MFCC (Mel-frequency cepstral coefficients) HMM state-of- the -art: GMM-HMM DNN 30 Hinton et al., deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE SP magazine, Nov. 2012 11/21/2012, The New York Times DNN NN NN Encoder-decoder paradigm Convolutional Neural Networks (CNN) CNN vs.

5 Fully-connected NN Restricted Boltzmann Machine (RBM) deep Belief Network (DBN) cuda-convnet MNIST NN =11 + = =tanh =max , 0 = + .. activation function NN =exp exp 1 2 1 2 softmax class probability input pattern NN =exp exp 1 2 1 2 softmax class probability input pattern 1, 1.

6 , , , = log cross entropy backpropagation NN =exp exp 1 2 1 2 softmax class probability input pattern GD = 1 ( ) momentum learning rate batch 1 stochastic GD mini batch Linearly separable patterns [Minsky-Papert] NN 1970 1980 1990 2000 2010 1960 Back-propagation [Rumelhart+] Sparse coding [Olfhausen-Field96] Convolutional NN [LeCun+89] Layerwise pretraining [Hinton+06] 1 2 3 Simple/complex cells [Hubel-Wiesel59] Perceptron [Rosenblatt57] NIPS2000 NN [Simard03]

7 Neo-cognitron [Fukushima80] DNN .. Bengio, Learning deep Architectures for AI, Foundations and Trends in Machine Learning, 2009 DNN Backpropagation [Hinton+06] GPU PC Linearly separable patterns [Minsky-Papert] NN 1970 1980 1990 2000 2010 1960 Back-propagation [Rumelhart+] Sparse coding [Olfhausen-Field96] Convolutional NN [LeCun+89] Layerwise pretraining [Hinton+06]

8 1 2 3 Simple/complex cells [Hubel-Wiesel59] Perceptron [Rosenblatt57] Neo-cognitron [Fukushima80] DNN NN NN Encoder-decoder paradigm Convolutional Neural Networks (CNN) CNN vs. fully-connected NN Restricted Boltzmann Machine (RBM) deep Belief Network (DBN) cuda-convnet MNIST DNN .. copy copy 1 2 3 3 2 1 3 2 1 4 init Encoder-decoder Autoencoder Restricted Boltzmann Machine (RBM).

9 = ( ) = ( ) min , ( ( ))2 .. state-of-the -art Convolutional Neural Network 80 [LeCun+89] ILSVRC2012 Supervision CNN w/o pretraining CIFAR10 Ciresan+12 CNN w/o pretraining NORB Ciresan+12 CNN w/o pretraining Le+12 TICA pretraining MNIST Ciresan+12 CNN w/o pretraining Ciresan+12 CNN w/o pretraining DNN NN NN Encoder-decoder paradigm Convolutional Neural Networks (CNN) CNN vs.

10 Fully-connected NN Restricted Boltzmann Machine (RBM) deep Belief Network (DBN) cuda-convnet MNIST Linearly separable patterns [Minsky-Papert] Convolutional Neural Network (CNN) 1970 1980 1990 2000 2010 1960 Back-propagation [Rumelhart+] Sparse coding [Olfhausen-Field96] Convolutional NN [LeCun+89] Layerwise pretraining [Hinton+06] 1 2 3 Simple/complex cells [Hubel-Wiesel59] Perceptron [Rosenblatt57] Neo-cognitron [Fukushima80] Convolutional Neural Network (CNN)