Lecture 3: Loss Functions and Optimization

1 Lecture 3: Loss Functions and Optimization Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 1 April 11, 2017 . Administrative Assignment 1 is released: Due Thursday April 20, 11:59pm on Canvas (Extending due date since it was released late). Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 2 April 11, 2017 . Administrative Check out Project Ideas on Piazza Schedule for Office hours is on the course website TA specialties are posted on Piazza Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 3 April 11, 2017 . Administrative Details about redeeming Google Cloud Credits should go out today;. will be posted on Piazza $100 per student to use for homeworks and projects Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 4 April 11, 2017 . Recall from last time: Challenges of recognition Viewpoint Illumination Deformation Occlusion This image by Umberto Salvagnin This image is CC0 public domain This image by jonsson is licensed is licensed under CC-BY under CC-BY Clutter Intraclass Variation This image is CC0 public domain This image is CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 5 April 11, 2017 .

2 Recall from last time: data-driven approach, kNN. 1-NN classifier 5-NN classifier train test train validation test Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 6 April 11, 2017 . Recall from last time: Linear Classifier f(x,W) = Wx + b Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 7 April 11, 2017 . Recall from last time: Linear Classifier TODO: 1. Define a loss function that quantifies our unhappiness with the scores across the training data. 2. Come up with a way of efficiently finding the parameters that minimize the loss function. ( Optimization ). Cat image by Nikita is licensed under CC-BY ; Car image is CC0 public domain; Frog image is in the public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 8 April 11, 2017 . Suppose: 3 training examples, 3 classes. With some W the scores are: cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 9 April 11, 2017 . Suppose: 3 training examples, 3 classes.

3 A loss function tells how With some W the scores are: good our current classifier is Given a dataset of examples Where is image and cat is (integer) label car Loss over the dataset is a sum of loss over examples: frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 10 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 11 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where Hinge loss . is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 12 April 11, 2017 . Suppose: 3 training examples, 3 classes.

4 Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 13 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - + 1). +max(0, - + 1). frog = max(0, ) + max(0, ). = + 0. Losses: = Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 14 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - + 1). +max(0, - + 1). frog = max(0, ) + max(0, ).

5 =0+0. Losses: 0 =0. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 15 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - ( ) + 1). +max(0, - ( ) + 1). frog = max(0, ) + max(0, ). = + Losses: 0 = Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 16 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Loss over full dataset is average: frog Losses: 0 L = ( + 0 + )/3. = Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 17 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q: What happens to frog loss if car scores change a bit?

6 Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 18 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q2: what is the frog min/max possible loss? Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 19 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q3: At initialization W. frog is small so all s 0. What is the loss? Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 20 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q4: What if the sum frog was over all classes?

7 (including j = y_i). Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 21 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q5: What if we used frog mean instead of sum? Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 22 April 11, 2017 . Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q6: What if we used frog Losses: 0 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 23 April 11, 2017 . Multiclass SVM Loss: Example code Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 24 April 11, 2017 . Suppose that we found a W such that L = 0.

8 Is this W unique? Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 25 April 11, 2017 . Suppose that we found a W such that L = 0. Is this W unique? No! 2W is also has L = 0! Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 26 April 11, 2017 . Suppose: 3 training examples, 3 classes. With some W the scores are: Before: = max(0, - + 1). +max(0, - + 1). = max(0, ) + max(0, ). =0+0. =0. cat With W twice as large: = max(0, - + 1). car +max(0, - + 1). = max(0, ) + max(0, ). frog =0+0. =0. Losses: 0. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 27 April 11, 2017 . Data loss: Model predictions should match training data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 28 April 11, 2017 . Data loss: Model predictions should match training data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 29 April 11, 2017 . Data loss: Model predictions should match training data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 30 April 11, 2017 .

9 Data loss: Model predictions should match training data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 31 April 11, 2017 . Data loss: Model predictions Regularization: Model should match training data should be simple , so it works on test data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 32 April 11, 2017 . Data loss: Model predictions Regularization: Model should match training data should be simple , so it works on test data Occam's Razor: Among competing hypotheses, the simplest is the best . William of Ockham, 1285 - 1347. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 33 April 11, 2017 . = regularization strength Regularization (hyperparameter). In common use: L2 regularization L1 regularization Elastic net (L1 + L2). Max norm regularization (might see later). Dropout (will see later). Fancier: Batch normalization, stochastic depth Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 34 April 11, 2017 . L2 Regularization (Weight Decay).

10 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 35 April 11, 2017 . L2 Regularization (Weight Decay). (If you are a Bayesian: L2. regularization also corresponds MAP inference using a Gaussian prior on W). Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 36 April 11, 2017 . Softmax Classifier (Multinomial Logistic Regression). cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 37 April 11, 2017 . Softmax Classifier (Multinomial Logistic Regression). scores = unnormalized log probabilities of the classes. cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 38 April 11, 2017 . Softmax Classifier (Multinomial Logistic Regression). scores = unnormalized log probabilities of the classes. where cat car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 39 April 11, 2017 . Softmax Classifier (Multinomial Logistic Regression). scores = unnormalized log probabilities of the classes. where cat Softmax function car frog Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 3 - 40 April 11, 2017 .