Lecture 3: Loss Functions and Optimization

1 Lecture 3: Loss Functions and Optimization Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 1 April 11, 2017. Administrative Assignment 1 is released: Due Thursday April 20, 11:59pm on Canvas (Extending due date since it was released late). Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 2 April 11, 2017. Administrative Check out Project Ideas on Piazza Schedule for Office hours is on the course website TA specialties are posted on Piazza Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 3 April 11, 2017.

2 Administrative Details about redeeming Google Cloud Credits should go out today;. will be posted on Piazza $100 per student to use for homeworks and projects Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 4 April 11, 2017. Recall from last time: Challenges of recognition Viewpoint Illumination Deformation Occlusion This image by Umberto Salvagnin This image is CC0 public domain This image by jonsson is licensed is licensed under CC-BY under CC-BY Clutter Intraclass Variation This image is CC0 public domain This image is CC0 public domain Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 5 April 11, 2017.

3 Recall from last time: data-driven approach, kNN. 1-NN classifier 5-NN classifier train test train validation test Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 6 April 11, 2017. Recall from last time: Linear Classifier f(x,W) = Wx + b Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 7 April 11, 2017. Recall from last time: Linear Classifier TODO: 1. Define a loss function that quantifies our unhappiness with the scores across the training data. 2. Come up with a way of efficiently finding the parameters that minimize the loss function.

4 ( Optimization ). Cat image by Nikita is licensed under CC-BY ; Car image is CC0 public domain; Frog image is in the public domain Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 8 April 11, 2017. Suppose: 3 training examples, 3 classes. With some W the scores are: cat car frog Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 9 April 11, 2017. Suppose: 3 training examples, 3 classes. A loss function tells how With some W the scores are: good our current classifier is Given a dataset of examples Where is image and cat is (integer) label car Loss over the dataset is a sum of loss over examples: frog Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 10 April 11, 2017.

5 Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 11 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where Hinge loss . is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 12 April 11, 2017.

6 Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car frog Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 13 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - + 1).

7 +max(0, - + 1). frog = max(0, ) + max(0, ). = + 0. Losses: = Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 14 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - + 1). +max(0, - + 1). frog = max(0, ) + max(0, ). =0+0. Losses: 0 =0. Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 15 April 11, 2017.

8 Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car = max(0, - ( ) + 1). +max(0, - ( ) + 1). frog = max(0, ) + max(0, ). = + Losses: 0 = Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 16 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Loss over full dataset is average: frog Losses: 0 L = ( + 0 + )/3.

9 = Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 17 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q: What happens to frog loss if car scores change a bit? Losses: 0 Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 18 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q2: what is the frog min/max possible loss?

10 Losses: 0 Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 19 April 11, 2017. Suppose: 3 training examples, 3 classes. Multiclass SVM loss: With some W the scores are: Given an example where is the image and where is the (integer) label, and using the shorthand for the scores vector: the SVM loss has the form: cat car Q3: At initialization W. frog is small so all s 0. What is the loss? Losses: 0 Fei-Fei Li & justin Johnson & Serena Yeung Lecture 3 - 20 April 11, 2017. Suppose: 3 training examples, 3 classes.