Lecture 7: Convolutional Neural Networks

1 Lecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 1 27 Jan 2016. Administrative A2 is due Feb 5 (next Friday). Project proposal due Jan 30 (Saturday). - ungraded, one paragraph - feel free to give 2 options, we can try help you narrow it - What is the problem that you will be investigating? Why is it interesting? - What data will you use? If you are collecting new datasets, how do you plan to collect them? - What method or algorithm are you proposing? If there are existing implementations, will you use them and how? How do you plan to improve or modify such implementations? - What reading will you examine to provide context and background? - How will you evaluate your results? Qualitatively, what kind of results do you expect ( plots or figures)?

2 Quantitatively, what kind of analysis will you use to evaluate and/or compare your results ( what performance metrics or statistical tests)? Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 2 27 Jan 2016. Mini-batch SGD. Loop: 1. Sample a batch of data 2. Forward prop it through the graph, get loss 3. Backprop to calculate the gradients 4. Update the parameters using the gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 3 27 Jan 2016. Parameter We covered: sgd, updates momentum, nag, adagrad, rmsprop, adam (not in this vis), we did not cover adadelta Image credits: Alec Radford Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 4 27 Jan 2016. Dropout Forces the network to have a redundant representation. has an ear X. has a tail is furry X cat score has claws mischievous X.

3 Look Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 5 27 Jan 2016. Convolutional Neural Networks [LeNet-5, LeCun 1980]. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 6 27 Jan 2016. A bit of history: Hubel & Wiesel, 1959. RECEPTIVE FIELDS OF SINGLE. NEURONES IN. THE CAT'S STRIATE CORTEX. 1962. RECEPTIVE FIELDS, BINOCULAR. INTERACTION. AND FUNCTIONAL ARCHITECTURE IN. THE CAT'S VISUAL CORTEX. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 7 27 Jan 2016. Hierarchical organization Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 8 27 Jan 2016. Convolutional Neural Networks (First without the brain stuff). Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 9 27 Jan 2016. Convolution Layer 32x32x3 image 32 height 32 width 3 depth Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 10 27 Jan 2016.

4 Convolution Layer 32x32x3 image 5x5x3 filter 32. Convolve the filter with the image slide over the image spatially, computing dot products . 32. 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 11 27 Jan 2016. Convolution Layer Filters always extend the full depth of the input volume 32x32x3 image 5x5x3 filter 32. Convolve the filter with the image slide over the image spatially, computing dot products . 32. 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 12 27 Jan 2016. Convolution Layer 32x32x3 image 5x5x3 filter 32. 1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image 32 ( 5*5*3 = 75-dimensional dot product + bias). 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 13 27 Jan 2016. Convolution Layer activation map 32x32x3 image 5x5x3 filter 32.

5 28. convolve (slide) over all spatial locations 32 28. 3 1. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 14 27 Jan 2016. consider a second, green filter Convolution Layer 32x32x3 image activation maps 5x5x3 filter 32. 28. convolve (slide) over all spatial locations 32 28. 3 1. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 15 27 Jan 2016. For example, if we had 6 5x5 filters, we'll get 6 separate activation maps: activation maps 32. 28. Convolution Layer 32 28. 3 6. We stack these up to get a new image of size 28x28x6! Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 16 27 Jan 2016. Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 28. CONV, ReLU. 6. 5x5x3. 32 filters 28. 3 6. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 17 27 Jan 2016.

6 Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions 32 28 24.. CONV, CONV, CONV, ReLU ReLU ReLU. 6 10. 5x5x3 5x5x6. 32 filters 28 filters 24. 3 6 10. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 18 27 Jan 2016. Preview [From recent Yann LeCun slides]. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 19 27 Jan 2016. Preview [From recent Yann LeCun slides]. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 20 27 Jan 2016. one filter =>. one activation map example 5x5 filters (32 total). We call the layer Convolutional because it is related to convolution of two signals: elementwise multiplication and sum of a filter and the signal (image). Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 21 27 Jan 2016. preview: Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 22 27 Jan 2016.

7 A closer look at spatial dimensions: activation map 32x32x3 image 5x5x3 filter 32. 28. convolve (slide) over all spatial locations 32 28. 3 1. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 23 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 24 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 25 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 26 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 27 Jan 2016.

8 A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter => 5x5 output 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 28 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter applied with stride 2. 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 29 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter applied with stride 2. 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 30 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter applied with stride 2. => 3x3 output! 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 31 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter applied with stride 3?

9 7. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 32 27 Jan 2016. A closer look at spatial dimensions: 7. 7x7 input (spatially). assume 3x3 filter applied with stride 3? 7 doesn't fit! cannot apply 3x3 filter on 7x7 input with stride 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 33 27 Jan 2016. N. Output size: (N - F) / stride + 1. F. N = 7, F = 3: F N. stride 1 => (7 - 3)/1 + 1 = 5. stride 2 => (7 - 3)/2 + 1 = 3. stride 3 => (7 - 3)/3 + 1 = :\. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 34 27 Jan 2016. In practice: Common to zero pad the border 0 0 0 0 0 0. input 7x7. 0 3x3 filter, applied with stride 1. 0 pad with 1 pixel border => what is the output? 0. 0. (recall:). (N - F) / stride + 1. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 35 27 Jan 2016.

10 In practice: Common to zero pad the border 0 0 0 0 0 0. input 7x7. 0 3x3 filter, applied with stride 1. 0 pad with 1 pixel border => what is the output? 0. 7x7 output! 0. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 36 27 Jan 2016. In practice: Common to zero pad the border 0 0 0 0 0 0. input 7x7. 0 3x3 filter, applied with stride 1. 0 pad with 1 pixel border => what is the output? 0. 7x7 output! 0. in general, common to see CONV layers with stride 1, filters of size FxF, and zero-padding with (F-1)/2. (will preserve size spatially). F = 3 => zero pad with 1. F = 5 => zero pad with 2. F = 7 => zero pad with 3. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 37 27 Jan 2016. Remember back to . 32x32 input convolved repeatedly with 5x5 filters shrinks volumes spatially!