Example: dental hygienist

Lecture 9: CNN Architectures

Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20171 Lecture 9:CNN ArchitecturesFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20172 AdministrativeA2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 session: Tue June 6, 12-3pmFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Last time: Deep learning frameworks3 Caffe (UC Berkeley)Torch (NYU / Facebook)Theano (U Montreal)TensorFlow (Google)Caffe2 (Facebook)PyTorch (Facebook)CNTK (Microsoft)Paddle (Baidu)MXNet (Amazon)Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of choice at AWSAnd Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20174(1)Easily build big computational graphs(2)Easily compute gradients in computational graphs(3)Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc)Last time: Deep learning frameworksFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20175 Last time: Deep learning frameworksModularized layers that define forward and backward passFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20176 Last time.

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 2 May 2, 2017 Administrative A2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu

Fullscreen Download

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Advertisement

Transcription of Lecture 9: CNN Architectures

1 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20171 Lecture 9:CNN ArchitecturesFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20172 AdministrativeA2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 session: Tue June 6, 12-3pmFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Last time: Deep learning frameworks3 Caffe (UC Berkeley)Torch (NYU / Facebook)Theano (U Montreal)TensorFlow (Google)Caffe2 (Facebook)PyTorch (Facebook)CNTK (Microsoft)Paddle (Baidu)MXNet (Amazon)Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of choice at AWSAnd Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20174(1)Easily build big computational graphs(2)Easily compute gradients in computational graphs(3)Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc)Last time: Deep learning frameworksFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20175 Last time: Deep learning frameworksModularized layers that define forward and backward passFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20176 Last time.

2 Deep learning frameworksDefine model architecture as a sequence of layersFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Today: CNN Architectures 7 Case (Network in Network)-Wide ResNet-ResNeXT-Stochastic Depth-DenseNet-FractalNet-SqueezeNetFei- Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20178 Review: LeNet-5[LeCun et al., 1998]Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride architecture is [CONV-POOL-CONV-POOL-FC-FC]Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 20179 Case Study: AlexNet[Krizhevsky et al. 2012]Architecture:CONV1 MAX POOL1 NORM1 CONV2 MAX POOL2 NORM2 CONV3 CONV4 CONV5 Max POOL3FC6FC7FC8 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201710 Case Study: AlexNet[Krizhevsky et al.]

3 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Q: what is the output volume size? Hint: (227-11)/4+1 = 55 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201711 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]Q: What is the total number of parameters in this layer?Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201712 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesFirst layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]Parameters: (11*11*3)*96 = 35 KFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201713 Case Study: AlexNet[Krizhevsky et al.

4 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2Q: what is the output volume size? Hint: (55-3)/2+1 = 27 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201714 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96Q: what is the number of parameters in this layer?Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201715 Case Study: AlexNet[Krizhevsky et al. 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 Second layer (POOL1): 3x3 filters applied at stride 2 Output volume: 27x27x96 Parameters: 0!Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201716 Case Study: AlexNet[Krizhevsky et al.

5 2012]Input: 227x227x3 imagesAfter CONV1: 55x55x96 After POOL1: Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201717 Case Study: AlexNet[Krizhevsky et al. 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201718 Case Study: AlexNet[Krizhevsky et al.

6 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Details/Retrospectives: - first use of ReLU- used Norm layers (not common anymore)- heavy data augmentation- dropout batch size 128- SGD Momentum Learning rate 1e-2, reduced by 10manually when val accuracy plateaus- L2 weight decay 5e-4- 7 CNN ensemble: -> Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201719 Case Study: AlexNet[Krizhevsky et al.

7 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)Historical note: Trained on GTX 580 GPU with only 3 GB of memory. Network spread across 2 GPUs, half the neurons (feature maps) on each GPU.[55x55x48] x 2 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201720 Case Study: AlexNet[Krizhevsky et al.

8 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)CONV1, CONV2, CONV4, CONV5: Connections only with feature maps on same GPUFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201721 Case Study: AlexNet[Krizhevsky et al. 2012]Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)CONV3, FC6, FC7, FC8.

9 Connections with all feature maps in preceding layer, communication across GPUsFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201722 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winnersFirst CNN-based winnerFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201723 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winnersZFNet: Improved hyperparameters over AlexNetFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201724 ZFNet[Zeiler and Fergus, 2013]AlexNet but:CONV1: change from (11x11 stride 4) to (7x7 stride 2)CONV3,4,5: instead of 384, 384, 256 filters use 512, 1024, 512 ImageNet top 5 error: -> : remake figureFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 201725 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winnersDeeper NetworksFei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Case Study.

10 VGGNet263x3 conv, 128 Pool3x3 conv, 643x3 conv, 64 Input3x3 conv, 128 Pool3x3 conv, 2563x3 conv, 256 Pool3x3 conv, 5123x3 conv, 512 Pool3x3 conv, 5123x3 conv, 512 PoolFC 4096FC 1000 SoftmaxFC 40963x3 conv, 5123x3 conv, 5123x3 conv, 384 Pool5x5 conv, 25611x11 conv, 96 InputPool3x3 conv, 3843x3 conv, 256 PoolFC 4096FC 4096 SoftmaxFC 1000 PoolInputPoolPoolPoolPoolSoftmax3x3 conv, 5123x3 conv, 5123x3 conv, 2563x3 conv, 2563x3 conv, 1283x3 conv, 1283x3 conv, 643x3 conv, 643x3 conv, 5123x3 conv, 5123x3 conv, 5123x3 conv, 5123x3 conv, 5123x3 conv, 512FC 4096FC 1000FC 4096[Simonyan and Zisserman, 2014]Small filters, Deeper networks 8 layers (AlexNet) -> 16 - 19 layers (VGG16 Net)Only 3x3 CONV stride 1, pad 1and 2x2 MAX POOL stride top 5 error in ILSVRC 13 (ZFNet)-> top 5 error in ILSVRC 14 AlexNetVGG16 VGG19 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Fei-Fei Li & Justin Johnson & Serena YeungLecture 9 -May 2, 2017 Case Study: VGGNet27[Simonyan and Zisserman, 2014]Q: Why use smaller filters?

Show more

Documents from same domain

NaveenAppiah SagarVare - Stanford University

NaveenAppiah SagarVare - Stanford University

cs231n.stanford.edu

NaveenAppiah Mechanical Engineering nappiahb@stanford.edu SagarVare Stanford ICME svare@stanford.edu ... the popular mobile game - Flappy Bird. It involves navi-gating a bird through a bunch of obstacles. Though, this ... the game emulator and learns to make good decisions over time. It is this simple learning framework and their

  Make, Games, Flappy, Naveenappiah

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 ...

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 2 - April 6, 2017 Administrative: Piazza For questions about midterm, poster session, projects,

  Li amp justin johnson amp serena yeung lecture

Attention and Transformers Lecture 11

Attention and Transformers Lecture 11

cs231n.stanford.edu

graph with shared weights h 0 f W h 1 f W h 2 f W h 3 x 3 y T ... Extract spatial features from a pretrained CNN Image Captioning using spatial features 11 CNN Features: H x W x D h 0 [START] Xu et al, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, ICML 2015 z 0,0 z 0,1 z 0,2 z 1,0 z 1,1 z 1,2 z 2,0 z 2,1 z ...

  Transformers, Attention, Graph, Spatial, Attention and transformers

CNNs for Face Detection and Recognition

CNNs for Face Detection and Recognition

cs231n.stanford.edu

development of object classification, localization and detec-tion techniques. 2.1. Sliding Window In the early development of face detection, researchers tended to treat it as a repetitive task of object classifica-tion, by imposing sliding windows and performing object classification with the neural networks on the window re-gion.

  Technique, Faces, Recognition, Object, Detection, For face detection and recognition

Vector, Matrix, and Tensor Derivatives

Vector, Matrix, and Tensor Derivatives

cs231n.stanford.edu

Erik Learned-Miller The purpose of this document is to help you learn to take derivatives of vectors, matrices, and higher order tensors (arrays with three dimensions or more), and to help you take ... At this point, we have reduced the original matrix equation (Equation 1) …

  Have, Learned

Lecture 14: Reinforcement Learning

Lecture 14: Reinforcement Learning

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 14 - May 23, 2017 Markov Decision Process 19 - Mathematical formulation of the RL problem - Markov property: Current state completely characterises the state of the

  Learning, Reinforcement, Reinforcement learning

Convolutional Neural Networks for Visual Recognition

Convolutional Neural Networks for Visual Recognition

cs231n.stanford.edu

Progressive GAN, Karras 2018. Models from Single RGB Images”, ECCV 2018 Beyond recognition: Segmentation, 2D/3D Generation. Fei-Fei Li, Ranjay Krishna, Danfei Xu Lecture 1 - 15 March 30, 2021 Scene Graphs Krishna et al., Visual Genome: Connecting Vision and Language using Crowdsourced Image Annotations, IJCV 2017

  Network, Visual, Recognition, Neural, Convolutional, Karar, Convolutional neural networks for visual recognition

Lecture 11: Detection and Segmentation

Lecture 11: Detection and Segmentation

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 1 May 10, 2017 Lecture 11: Detection and Segmentation

  Detection, Segmentation

Lecture 13: Generative Models

Lecture 13: Generative Models

cs231n.stanford.edu

Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 13 - May 18, 2017 Generative Models 17 Training data ~ p data (x) Generated samples ~ p model (x) Want to learn p

  Generative

Lecture 10: Recurrent Neural Networks

Lecture 10: Recurrent Neural Networks

cs231n.stanford.edu

image -> sequence of words. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 10 - 13 May 4, 2017 Recurrent Neural Networks: Process Sequences e.g. Sentiment Classification sequence of words -> sentiment. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 10 - 14 May 4, 2017

  Image, Recurrent

Related documents

Final Guidelines - Europa

Final Guidelines - Europa

esas-joint-committee.europa.eu

THE RISK FACTORS GUIDELINES . JC 2017 37 04/01/2018 . Final Guidelines Joint Guidelines under Articles 17 and 18(4) of Directive (EU) 2015/849 on …

The Limits of Quantum Computers - University of Virginia ...

The Limits of Quantum Computers - University of Virginia ...

www.cs.virginia.edu

ity. Given any holy grail of mathematics—such as Goldbach’s conjecture or the Riemann hy-pothesis, both of which have resisted resolution for well over a century—we could simply ask our computer to search through all possible proofs

Price setting and price regulation in health care

Price setting and price regulation in health care

www.oecd.org

Price setting and price regulation in health care: lessons for advancing Universal Health Coverage Sarah L Barber, Luca Lorenzoni, Paul Ong ISBN 978-92-4-151592-4 (WHO)