Lecture 11: Generative Models - Stanford University

1 Lecture 11: Generative Models Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 1 May 9, 2019. Administrative A3 is out. Due May 22. Milestone is due next Wednesday. Read Piazza post for milestone requirements. Need to Finish data preprocessing and initial results by then. Don't discuss exam yet since people are still taking it. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 -2 May 9, 2019. Overview Unsupervised Learning Generative Models PixelRNN and PixelCNN. Variational Autoencoders (VAE). Generative Adversarial Networks (GAN). Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 3 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Data: (x, y). x is data, y is label Goal: Learn a function to map x -> y Examples: Classification, regression, object detection, semantic segmentation, image captioning, etc.

2 Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 4 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Data: (x, y). x is data, y is label Cat Goal: Learn a function to map x -> y Examples: Classification, regression, object detection, Classification semantic segmentation, image captioning, etc. This image is CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 5 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Data: (x, y). x is data, y is label Goal: Learn a function to map x -> y Examples: Classification, DOG, DOG, CAT. regression, object detection, semantic segmentation, image Object Detection captioning, etc. This image is CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 6 May 9, 2019.

3 Supervised vs Unsupervised Learning Supervised Learning Data: (x, y). x is data, y is label Goal: Learn a function to map x -> y GRASS, CAT, Examples: Classification, TREE, SKY. regression, object detection, semantic segmentation, image Semantic Segmentation captioning, etc. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 7 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Data: (x, y). x is data, y is label Goal: Learn a function to map x -> y A cat sitting on a suitcase on the floor Examples: Classification, regression, object detection, Image captioning semantic segmentation, image captioning, etc. Caption generated using neuraltalk2. Image is CC0 Public domain. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 8 May 9, 2019.

4 Supervised vs Unsupervised Learning Unsupervised Learning Data: x Just data, no labels! Goal: Learn some underlying hidden structure of the data Examples: Clustering, dimensionality reduction, feature learning, density estimation, etc. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 9 May 9, 2019. Supervised vs Unsupervised Learning Unsupervised Learning Data: x Just data, no labels! Goal: Learn some underlying hidden structure of the data Examples: Clustering, K-means clustering dimensionality reduction, feature learning, density estimation, etc. This image is CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 10 May 9, 2019. Supervised vs Unsupervised Learning Unsupervised Learning Data: x Just data, no labels!

5 Goal: Learn some underlying hidden structure of the data 3-d 2-d Examples: Clustering, Principal Component Analysis dimensionality reduction, feature (Dimensionality reduction). learning, density estimation, etc. This image from Matthias Scholz is CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 11 May 9, 2019. Supervised vs Unsupervised Learning Unsupervised Learning Data: x Just data, no labels! Goal: Learn some underlying hidden structure of the data Examples: Clustering, dimensionality reduction, feature Autoencoders learning, density estimation, etc. (Feature learning). Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 12 May 9, 2019. Supervised vs Unsupervised Learning Unsupervised Learning Data: x Figure copyright Ian Goodfellow, 2016.

6 Reproduced with permission. Just data, no labels! 1-d density estimation Goal: Learn some underlying hidden structure of the data Examples: Clustering, dimensionality reduction, feature 2-d density estimation learning, density estimation, etc. 2-d density images left and right are CC0 public domain Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 13 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Unsupervised Learning Data: (x, y) Data: x x is data, y is label Just data, no labels! Goal: Learn a function to map x -> y Goal: Learn some underlying hidden structure of the data Examples: Classification, regression, object detection, Examples: Clustering, semantic segmentation, image dimensionality reduction, feature captioning, etc.

7 Learning, density estimation, etc. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 14 May 9, 2019. Supervised vs Unsupervised Learning Supervised Learning Unsupervised Learning Training data is cheap Data: (x, y) Data: x Holy grail: Solve x is data, y is label Just data, no labels! unsupervised learning => understand structure of visual world Goal: Learn a function to map x -> y Goal: Learn some underlying hidden structure of the data Examples: Classification, regression, object detection, Examples: Clustering, semantic segmentation, image dimensionality reduction, feature captioning, etc. learning, density estimation, etc. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 15 May 9, 2019. Generative Models Given training data, generate new samples from same distribution Training data ~ pdata(x) Generated samples ~ pmodel(x).

8 Want to learn pmodel(x) similar to pdata(x). Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 16 May 9, 2019. Generative Models Given training data, generate new samples from same distribution Training data ~ pdata(x) Generated samples ~ pmodel(x). Want to learn pmodel(x) similar to pdata(x). Addresses density estimation, a core problem in unsupervised learning Several flavors: - Explicit density estimation: explicitly define and solve for pmodel(x). - Implicit density estimation: learn model that can sample from pmodel(x) w/o explicitly defining it Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 17 May 9, 2019. Why Generative Models ? - Realistic samples for artwork, super-resolution, colorization, etc. - Generative Models of time-series data can be used for simulation and planning (reinforcement learning applications!)

9 - Training Generative Models can also enable inference of latent representations that can be useful as general features FIgures from L-R are copyright: (1) Alec Radford et al. 2016; (2) Phillip Isola et al. 2017. Reproduced with authors permission (3) BAIR Blog. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 18 May 9, 2019. Taxonomy of Generative Models Direct GAN. Generative Models Explicit density Implicit density Markov Chain Tractable density Approximate density GSN. Fully Visible Belief Nets - NADE. - MADE Variational Markov Chain - PixelRNN/CNN. Variational Autoencoder Boltzmann Machine - NICE / RealNVP. - Glow Figure copyright and adapted from Ian Goodfellow, Tutorial on Generative Adversarial Networks, 2017. - Ffjord Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 19 May 9, 2019.

10 Taxonomy of Generative Models Direct Today: discuss 3 most GAN. popular types of Generative Generative Models Models today Explicit density Implicit density Markov Chain Tractable density Approximate density GSN. Fully Visible Belief Nets - NADE. - MADE Variational Markov Chain - PixelRNN/CNN. Variational Autoencoder Boltzmann Machine - NICE / RealNVP. - Glow Figure copyright and adapted from Ian Goodfellow, Tutorial on Generative Adversarial Networks, 2017. - Ffjord Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 20 May 9, 2019. PixelRNN and PixelCNN. Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 21 May 9, 2019. Fully visible belief network Explicit density model Use chain rule to decompose likelihood of an image x into product of 1-d distributions: Likelihood of Probability of i'th pixel value image x given all previous pixels Then maximize likelihood of training data Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 11 - 22 May 9, 2019.