Introduction - Deep Learning

1 Introduction Lecture slides for Chapter 1 of deep Learning Ian Goodfellow 2016-09-26. Representations Matter APTER 1. Introduction . Cartesian coordinates Polar coordinates y . x r Figure suppose we want to separate ure : Example of di erent representations: (Goodfellow 2016). Depth: Repeated Composition CHAPTER 1. Introduction . Output CAR PERSON ANIMAL. (object identity). 3rd hidden layer (object parts). 2nd hidden layer (corners and contours). 1st hidden layer (edges). Visible layer (input pixels). Figure : Illustration of a deep Learning model. It is di cult for a computer to understand Figure the meaning of raw sensory input data, such as this image represented as a collection (Goodfellow 2016). Computational Graphs CHAPTER 1. Introduction . Element Element Set Set +. +.

2 Logistic Regression Logistic Regression w1 x1 w2 x2 w x Figure : Illustration of computational graphs mapping an input to an output where Figure each node performs an operation. Depth is length of the longest path from(Goodfellow input 2016)to Machine Learning and AI. deep Learning Example: Shallow Example: Example: Example: autoencoders Logistic Knowledge MLPs regression bases Representation Learning Machine Learning AI. Figure Figure : A Venn diagram showing how deep Learning is a kind of representation Learning , (Goodfellow 2016). CHAPTER 1. Introduction . Learning Multiple Components Figure Output Mapping from Output Output features Additional Mapping from Mapping from layers of more Output features features abstract features Hand- Hand- Simple designed designed Features features program features Input Input Input Input deep Classic Learning Rule-based machine systems Representation Learning (Goodfellow 2016).

3 Learning Organization of the Book CHAPTER 1. Introduction . Figure 1. Introduction Part I: Applied Math and Machine Learning Basics 3. Probability and 2. Linear Algebra Information theory 4. Numerical 5. Machine Learning Computation Basics Part II: deep Networks: Modern Practices 6. deep Feedforward Networks 7. Regularization 8. Optimization 9. CNNs 10. RNNs 11. Practical 12. Applications Methodology Part III: deep Learning Research 13. Linear Factor 15. Representation 14. Autoencoders Models Learning 16. Structured 17. Monte Carlo Probabilistic Models Methods 18. Partition 19. Inference Function 20. deep Generative Models (Goodfellow 2016). Figure : The high-level organization of the book. An arrow from one chapter to another Historical Waves Frequency of Word or Phrase cybernetics (connectionism + neural networks).

4 1940 1950 1960 1970 1980 1990 2000. Year ure : The figure shows two of Figure the historical waves of artificial neural (Goodfellow 2016). Historical Trends: Growing Datasets 109. Dataset size (number examples). 108 Canadian Hansard WMT Sports-1M. 107 ImageNet10k 106 Public SVHN. 105 Criminals ImageNet ILSVRC 2014. 104. MNIST CIFAR-10. 103. 102 T vs. G vs. F Rotated T vs. C. Iris 101. 100. 1900 1950 1985 2000 2015. Year ure : Dataset sizes have increased greatly over time. In the early 1900s, statistician died datasets using hundreds or thousands of manually compiled measurements (Garson 0; Gosset, 1908; Anderson, 1935; Fisher, 1936). In the 1950s through 1980s, the pioneer iologically inspired machine learningFigure with small, synthetic datasets, often worked (Goodfellow suc 2016).

5 CHAPTER 1. Introduction . The MNIST Dataset Figure Figure : Example inputs from the MNIST dataset. The NIST stands for National (Goodfellow 2016). Connections per Neuron 104 Human 6 Cat Connections per neuron 9 7. 4. 103 Mouse 2. 10. 5. 8. 102 Fruit fly 3. 1. 101. 1950 1985 2000 2015. Year Figure : Initially, the number of connections between neurons in artificial neura networks was limited by hardware capabilities. Today, the number of connections between Figure neurons is mostly a design consideration. artificial neural networks have(Goodfellow nearly a 2016). Number of Neurons Number of neurons (logarithmic scale). 1011 Human 1010. 17 20. 109 16 19 Octopus 108 14 18. 107 11 Frog 106 8. 105 3 Bee Ant 104. 103 Leech 13. 102. 101 1 2 12 15 Roundworm 6 9. 100 5 10. 10 1 4 7.

6 10 2 Sponge 1950 1985 2000 2015 2056. Year ure : Since the Introduction of hidden units, artificial neural networks have doub ize roughly every years. Biological neural network sizes from Wikipedia (2015. 1. Perceptron (Rosenblatt, 1958, 1962) Figure (Goodfellow 2016). Solving Object Recognition ILSVRC classification error rate 2010 2011 2012 2013 2014 2015. Year gure : Since deep networks reached the scale necessary to compete in the Ima arge Scale Visual Recognition Challenge, they have consistently won the compe ery year, and yielded lower and lower Figureerror rates each time. Data from(Goodfellow Russak 2016).)