Example: bachelor of science

Lecture 9: CNN Architectures

Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20171 Lecture 9:CNN ArchitecturesFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20172 AdministrativeA2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 session: Tue June 6, 12-3pmFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Last time: Deep learning frameworks3 Caffe (UC Berkeley)Torch (NYU / Facebook)Theano (U Montreal)TensorFlow (Google)Caffe2 (Facebook)PyTorch (Facebook)CNTK (Microsoft)Paddle (Baidu)MXNet (Amazon)Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of choice at AWSAnd Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20174(1)Easily build big computational graphs(2)Easily compute gradients in computational graphs(3)Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc)Last time.

ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winners Deeper Networks. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - May 2, 2017 Case Study: VGGNet 26 3x3 conv, 128 Pool 3x3 conv, 64 3x3 conv, 64 Input 3x3 conv, 128 Pool 3x3 conv, 256 3x3 conv, 256 Pool 3x3 conv, 512 3x3 conv, 512 Pool 3x3 conv, 512 3x3 conv, 512 Pool FC 4096

Fullscreen Download

Tags:

  Visual

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Advertisement

Transcription of Lecture 9: CNN Architectures

1 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20171 Lecture 9:CNN ArchitecturesFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20172 AdministrativeA2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 session: Tue June 6, 12-3pmFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Last time: Deep learning frameworks3 Caffe (UC Berkeley)Torch (NYU / Facebook)Theano (U Montreal)TensorFlow (Google)Caffe2 (Facebook)PyTorch (Facebook)CNTK (Microsoft)Paddle (Baidu)MXNet (Amazon)Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of choice at AWSAnd Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20174(1)Easily build big computational graphs(2)Easily compute gradients in computational graphs(3)Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc)Last time.

2 Deep learning frameworksFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20175 Last time: Deep learning frameworksModularized layers that define forward and backward passFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20176 Last time: Deep learning frameworksDefine model architecture as a sequence of layersFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Today: CNN Architectures 7 Case (Network in Network)-Wide ResNet-ResNeXT-Stochastic Depth-DenseNet-FractalNet-SqueezeNetFei- Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20178 Review: LeNet-5[LeCun et al., 1998]Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride architecture is [CONV-POOL-CONV-POOL-FC-FC]Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20179 Case Study: AlexNet[Krizhevsky et al.]

3 2012]Architecture:CONV1 MAX POOL1 NORM1 CONV2 MAX POOL2 NORM2 CONV3 CONV4 CONV5 Max POOL3FC6FC7FC8 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201710 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Q: what is the output volume size? Hint: (227-11)/4+1 = 55 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201711 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]Q: What is the total number of parameters in this layer?

4 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201712 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]Parameters: (11*11*3)*96 = 35 KFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201713 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2Q: what is the output volume size? Hint: (55-3)/2+1 = 27 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201714 Case Study: AlexNet[Krizhevsky et al.]

5 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96Q: what is the number of parameters in this layer?Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201715 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96 Parameters: 0!Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201716 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 After POOL1: Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201717 Case Study: AlexNet[Krizhevsky et al.

6 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201718 Case Study: AlexNet[Krizhevsky et al.

7 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Details/Retrospectives: - first use of ReLU- used Norm layers (not common anymore)- heavy data augmentation- dropout batch size 128- SGD Momentum Learning rate 1e-2, reduced by 10manually when val accuracy plateaus- L2 weight decay 5e-4- 7 CNN ensemble: -> Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201719 Case Study: AlexNet[Krizhevsky et al.

8 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Historical note: Trained on GTX 580 GPU with only 3 GB of memory.

9 Network spread across 2 GPUs, half the neurons (feature maps) on each GPU.[55x55x48] x 2 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201720 Case Study: AlexNet[Krizhevsky et al. 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)CONV1, CONV2, CONV4, CONV5.

10 Connections only with feature maps on same GPUFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201721 Case Study: AlexNet[Krizhevsky et al. 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)CONV3, FC6, FC7, FC8.

Show more

Documents from same domain

Attention and Transformers Lecture 11

Attention and Transformers Lecture 11

cs231n.stanford.edu

graph with shared weights h 0 f W h 1 f W h 2 f W h 3 x 3 y T ... Extract spatial features from a pretrained CNN Image Captioning using spatial features 11 CNN Features: H x W x D h 0 [START] Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 z 0,0 z 0,1 z 0,2 z 1,0 z 1,1 z 1,2 z 2,0 z 2,1 z ...

  Transformers, Attention, Graph, Spatial, Attention and transformers

Convolutional Neural Networks for Visual Recognition

Convolutional Neural Networks for Visual Recognition

cs231n.stanford.edu

Progressive GAN, Karras 2018. Models from Single RGB Images”, ECCV 2018 Beyond recognition: Segmentation, 2D/3D Generation. Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 1 - 15 March 30, 2021 Scene Graphs Krishna et al., Visual Genome: Connecting Vision and Language using Crowdsourced Image Annotations, IJCV 2017

  Network, Visual, Recognition, Neural, Convolutional, Karar, Convolutional neural networks for visual recognition

NaveenAppiah SagarVare - Stanford University

NaveenAppiah SagarVare - Stanford University

cs231n.stanford.edu

NaveenAppiah Mechanical Engineering nappiahb@stanford.edu SagarVare Stanford ICME svare@stanford.edu ... the popular mobile game - Flappy Bird. It involves navi-gating a bird through a bunch of obstacles. Though, this ... the game emulator and learns to make good decisions over time. It is this simple learning framework and their

  Make, Games, Flappy, Naveenappiah

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 - April 6, 2017 Administrative: Piazza For questions about midterm, poster session, projects,

  Li amp justin johnson amp serena yeung lecture

CNNs for Face Detection and Recognition

CNNs for Face Detection and Recognition

cs231n.stanford.edu

development of object classification, localization and detec-tion techniques. 2.1. Sliding Window In the early development of face detection, researchers tended to treat it as a repetitive task of object classifica-tion, by imposing sliding windows and performing object classification with the neural networks on the window re-gion.

  Technique, Faces, Recognition, Object, Detection, For face detection and recognition

Vector, Matrix, and Tensor Derivatives

Vector, Matrix, and Tensor Derivatives

cs231n.stanford.edu

Erik Learned-Miller The purpose of this document is to help you learn to take derivatives of vectors, matrices, and higher order tensors (arrays with three dimensions or more), and to help you take ... At this point, we have reduced the original matrix equation (Equation 1) …

  Have, Learned

Lecture 14: Reinforcement Learning

Lecture 14: Reinforcement Learning

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 14 - May 23, 2017 Markov Decision Process 19 - Mathematical formulation of the RL problem - Markov property: Current state completely characterises the state of the

  Learning, Reinforcement, Reinforcement learning

Lecture 11: Detection and Segmentation

Lecture 11: Detection and Segmentation

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 1 May 10, 2017 Lecture 11: Detection and Segmentation

  Detection, Segmentation

Lecture 13: Generative Models

Lecture 13: Generative Models

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 13 - May 18, 2017 Generative Models 17 Training data ~ p data (x) Generated samples ~ p model (x) Want to learn p

  Generative

Lecture 10: Recurrent Neural Networks

Lecture 10: Recurrent Neural Networks

cs231n.stanford.edu

image -> sequence of words. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 10 - 13 May 4, 2017 Recurrent Neural Networks: Process Sequences e.g. Sentiment Classification sequence of words -> sentiment. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 10 - 14 May 4, 2017

  Image, Recurrent