Lecture 3: Neural Network Basics & Architecture Design

1 Lecture 3: Neural Network Basics & Architecture DesignXiangyu ZhangFace++ RecognitionA fundamental task in computer vision Classification Object Detection Semantic Segmentation Instance Segmentation Key point Detection Recognition Difficult?Multiple ObjectsPoseOcclusionInter-classSimilarit yAny Silver Bullet? Deep Neural NetworksOutline Neural Network Basics Architecture DesignPART 1: Neural Network Basics Motivation Deep Neural networks Convolutional Neural Networks (CNNs)** Special thanks Marc'AurelioRanzatofor the tutorial Large-Scale Visual RecognitionWith Deep Learning in CVPR 2013. All pictures are owned by the 1: Neural Network Basics Motivation Deep Neural networks Convolutional Neural Networks (CNNs)Features for RecognitionNonlinear Features vs.

2 Linear ClassifiersFeature extractor should be nonlinear!Learning Non-Linear Features Q: which class of non-linear functions shall we consider?Shallow or DeepShallowDeepLinear CombinationDrawbacks:Exponential number of templates required! Kernel learning Boosting ..CompositionMain Idea of Deep LearningConcepts Reuse in Deep LearningZeilerM D, Fergus R. Visualizing and understanding convolutional networksConcepts Reuse in Deep Learning (cont d)ZeilerM D, Fergus R. Visualizing and understanding convolutional networksConcepts Reuse in Deep Learning (cont d)Efficiency: intermediate concepts can be re-usedDeep Learning FrameworkA problem:Optimizationis difficult: non-convex, non-linear systemDeep Learning Framework (cont d)Deep Learning Framework (cont d)Summary: Key Ideas of Deep Learning We need nonlinear system We need to learn it from data Build feature hierarchies (function composition) End-to-end learningPART 1: Neural Network Basics Motivation Deep Neural networks Convolutional Neural Networks (CNNs)How to Build Deep Network ?

3 Neuron or Layer DesignShallow Cases Linear Case: SVMS hallow Cases (cont d) Linear Case: Logistic RegressionLinear transformation + nonlinear activationNeuron DesignSingle Neuron:Linear Projection + Nonlinear ActivationDeep Neuron NetworkDeep Neural Network (cont d)Gradient-based Training For each Parameters (Optimization) Forward Propagation (FPROP)Forward Propagation (FPROP)This is the typical processing at test time. At training time, we need to compute an error measure and tune the parameters to decrease the FunctionLoss FunctionQ: how to tune the parameters to decrease the loss?A: If loss is ( ) differentiable we can compute gradients. We can use chain-rule, back-propagation, to compute the gradients parameters at the lower Propagation (BPROP)Backward Propagation (BPROP) (cont d)Backward Propagation (BPROP) (cont d)Optimization Stochastic Gradient Descent (on mini-batches): Stochastic Gradient Descent with Momentum:Summary: Key Ideas of Deep Neural Networks Neural Net = stack of feature detectors F-Prop / B-Prop Learning by SGDPART 1: Neural Network Basics Motivation Deep Neural networks Convolutional Neural Networks (CNNs) Deep Neural Networks on Images How to apply a Neural Network on 2D or 3D inputs?

4 Fully-connected NetLocally-connected NetSTATIONARITY? Statistics are similar at different locations (translation invariance)Convolutional NetConvolutional Net (cont d)Convolutional Net (cont d)Convolutional Net (cont d)Convolutional LayerConvolutional Layer (cont d)Summary: Key Ideas of Convolutional Nets A standard Neural net applied to images: scales quadratically with the size of the input does not leverage stationarity Solution: connect each hidden unit to a small patch of the input share the weight across hidden units This is called: convolutional Layers Over the years, some new modules have proven to be very effective when plugged into conv-nets:Pooling LayerPooling LayerLocal Contrast Normalization LayerTypical ArchitectureQ.

5 Where is the nonlinearity?Typical Architecture (cont d)Conv Architecture Example (AlexNet)Krizhevskyet al. ImageNet Classification with deep CNNs NIPS 2012 Convolutional Nets: Training All layers are differentiable ( ). We can use standard back-propagation. Algorithm:Given a small UPDATES ummary: Key Ideas of Conv Nets Conv. Nets have special layers like: pooling, and local contrast normalization Back-propagation can still be applied. These layers are useful to: reduce computational burden increase invariance ease the optimizationPART 2: Architecture Design Overview Structure Design Layer Design Architecture for special tasksPART 2: Architecture Design Overview Structure Design Layer Design Architecture for special tasksArchitecture Design What?

6 Network topology Layer functions Hyper-parameters Optimization Why? Difficult to determine the optimal structures Requirements of different applications, datasets or limitations Architecture Design (cont d) How? Manually Automatically Objective Representation capability Robustness, anti-overfitting Computation or parameter efficiency Ease of accuracy, less complexityPART 2: Architecture Design Overview Structure Design Layer Design Architecture for special tasksBenchmark: ImageNet Dataset 1K classes (for ILSVRC competition) + training images, 50K validation images, 100K test images ILSVRC competitionDifficulty Fine-grained classes Large variation Costly trainingBenchmark: ImageNet Dataset 1K classes (for ILSVRC competition) + training images, 50K validation images, 100K test images ILSVRC competitionDifficulty Fine-grained classes Large variation Costly trainingWalker houndBeagleEnglish foxhound?

7 Benchmark: ImageNet Dataset 1K classes (for ILSVRC competition) + training images, 50K validation images, 100K test images ILSVRC competitionDifficulty Fine-grained classes Large variation Costly trainingBenchmark: ImageNet Dataset 1K classes (for ILSVRC competition) + training images, 50K validation images, 100K test images ILSVRC competitionDifficulty Fine-grained classes Large variation Costly trainingRecent Nets ImageNet Classification Scores8 layers19 layers152 layers8 layers22 layersAlexNetKrizhevskyA, SutskeverI, Hinton G E. Imagenetclassification with deep convolutional Neural networksVGGNetSimonyanK, Zisserman A. Very deep convolutional networks for large-scale image recognitionGoogleNetSzegedyC, Liu W, Jia Y, et al.

8 Going deeper with convolutionsDeep Residual Network Easy to optimize Enable very deep structures--Over 100 layers for ImageNet modelHe K, Zhang X, Ren S, et al. Deep residual learning for image recognitionDeep Residual Network (cont d) Bottleneck Design Increasing depth, less complexityHe K, Zhang X, Ren S, et al. Deep residual learning for image recognitionXceptionCholletF. Xception: Deep Learning with DepthwiseSeparable ConvolutionsResNeXtXie S, GirshickR, Doll rP, et al. Aggregated residual transformations for deep Neural networksShuffleNetZhang X, Zhou X, Lin M, et al. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile DevicesDensely Connected Convolutional NetworksHuang G, Liu Z, Weinberger K Q, et al.

9 Densely connected convolutional networksSqueeze-and- excitation NetworksHu J, Shen L, Sun G. squeeze -and- excitation NetworksSummary: Ideas of Structure Design Deeper and wider Ease of optimization Multi-path Design Residual path Sparse connectionPART 2: Architecture Design Overview Structure Design Layer Design Architecture for special tasksSpatial Pyramid PoolingHe K, Zhang X, Ren S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognitionBatch NormalizationBatch normalization: Accelerating deep Network training by reducing internal covariate shiftParametric RectifiersHe K, Zhang X, Ren S, et al. Delving deep into rectifiers: Surpassing human-level performance on imagenetclassificationBilinear CNNsLin T Y, RoyChowdhuryA, MajiS.

10 Bilinear cnnmodels for fine-grained visual recognitionPART 2: Architecture Design Overview Structure Design Layer Design Architecture for special tasksDeepfaceTaigmanY, Yang M, RanzatoM A, et al. Deepface: Closing the gap to human-level performance in face verificationGlobal Convolutional NetworksPeng C, Zhang X, Yu G, et al. Large Kernel Matters--Improve Semantic Segmentation by Global Convolutional NetworkHourglass NetworksNewell A, Yang K, Deng J. Stacked hourglass networks for human pose estimationSummary: Trends on Architecture Design Effectiveness and efficiency Task & data specific ML & optimization perspective Insight & motivation drivenThanks