Attention and Transformers Lecture 11

1 Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20211 Lecture 11: Attention and TransformersFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20212 Administrative: Midterm-Midterm was this Tuesday-We will be grading this week and you should have grades by next Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20213 Administrative: Assignment 3-A3 is due Friday May 25th, 11:59pm Lots of applications of ConvNets Also contains an extra credit notebook, which is worth an additional 5% of the A3 grade. Extra credit will not be used when curving the class Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20214 Last Time: Recurrent Neural NetworksFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20215 Last Time: Variable length computation graph with shared x2x1 WhTy3y2y1L1L2L3 LTLFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Let's jump to Lecture 10 - slide 436 Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20217 Today's Agenda:- Attention with RNNs-In Computer Vision-In NLP-General Attention Layer-Self- Attention -Positional encoding-Masked Attention -Multi-head Attention -TransformersFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 20218 Today's Agenda.

2 - Attention with RNNs-In Computer Vision-In NLP-General Attention Layer-Self- Attention -Positional encoding-Masked Attention -Multi-head Attention -TransformersFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features9 CNNF eatures: H x W x DXu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 Input: Image IOutput: Sequence y = y1, y2,.., yTFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features10 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 MLPE ncoder: h0 = fW(z)where z is spatial CNN featuresfW(.) is an MLPI nput: Image IOutput: Sequence y = y1, y2,.., yTFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features11 CNNF eatures: H x W x Dh0[START]Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 y0h1[START]y1personMLPE ncoder: h0 = fW(z)where z is spatial CNN featuresfW(.)

3 Is an MLPI nput: Image IOutput: Sequence y = y1, y2,.., yTDecoder: yt = gV(yt-1, ht-1, c)where context vector c is often c = h0cFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features12 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 MLPE ncoder: h0 = fW(z)where z is spatial CNN featuresfW(.) is an MLPI nput: Image IOutput: Sequence y = y1, y2,.., yTDecoder: yt = gV(yt-1, ht-1, c)where context vector c is often c = h0h0[START]y0h1[START]y1h2y2y1personpers onwearingcFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features13 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 MLPE ncoder: h0 = fW(z)where z is spatial CNN featuresfW(.)

4 Is an MLPI nput: Image IOutput: Sequence y = y1, y2,.., yTDecoder: yt = gV(yt-1, ht-1, c)where context vector c is often c = h0h0[START]y0h1[START]y1h2y2y1h3y3y2pers onwearingpersonwearinghatcFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features14 CNNF eatures: H x W x Dh0[START]Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 y0h1[START]y1h2y2y1h3y3y2personwearingha th4y4y3personwearinghat[END]MLPcEncoder: h0 = fW(z)where z is spatial CNN featuresfW(.) is an MLPI nput: Image IOutput: Sequence y = y1, y2,.., yTDecoder: yt = gV(yt-1, ht-1, c)where context vector c is often c = h0 Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning using spatial features15 CNNF eatures: H x W x Dh0[START]Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 y0h1[START]y1h2y2y1h3y3y2personwearingha th4y4y3personwearinghat[END]MLPP roblem: Input is "bottlenecked" through c-Model needs to encode everything it wants to say within cThis is a problem if we want to generate really long descriptions?

5 100s of words longcFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention16 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 Attention idea: New context vector at every time context vector will attend to different image regionsgif sourceAttention Saccades in humansFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention17 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 e1,0,0e1,0,1e1,0,2e1,1,0e1,1,1e1,1,2e1,2 ,0e1,2,1e1,2,2 Alignment scores: H x W Compute alignments scores (scalars):fatt(.) is an MLPFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention18 CNNF eatures: H x W x Dh0Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 e1,0,0e1,0,1e1,0,2e1,1,0e1,1,1e1,1,2e1,2 ,0e1,2,1e1,2,2a1,0,0a1,0,1a1,0,2a1,1,0a1 ,1,1a1,1,2a1,2,0a1,2,1a1,2,2 Alignment scores: H x W Attention : H x W Normalize to get Attention weights:0 < at, i, j < 1, Attention values sum to 1 Compute alignments scores (scalars):fatt(.)

6 Is an MLPFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention19 CNNF eatures: H x W x Dh0c1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0z0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 e1,0,0e1,0,1e1,0,2e1,1,0e1,1,1e1,1,2e1,2 ,0e1,2,1e1,2,2a1,0,0a1,0,1a1,0,2a1,1,0a1 ,1,1a1,1,2a1,2,0a1,2,1a1,2,2 Alignment scores: H x W Attention : H x W XCompute alignments scores (scalars):fatt(.) is an MLPC ompute context vector:Normalize to get Attention weights:0 < at, i, j < 1, Attention values sum to 1 Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNEach timestep of decoder uses a different context vector that looks at different parts of the input imageImage Captioning with RNNs & Attention20 CNNF eatures: H x W x Dh0c1y0h1[START]y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0personz0,1z0,2z1,0z1,1z1,2z2,0z2 ,1z2,2 Decoder: yt = gV(yt-1, ht-1, ct)New context vector at every time stepFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention21 CNNF eatures: H x W x Dh0c1y0h1[START]y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0personz0,1z0,2z1,0z1,1z1,2z2,0z2 ,1z2,2 Decoder.

7 Yt = gV(yt-1, ht-1, ct)New context vector at every time stepe1,0,0e1,0,1e1,0,2e1,1,0e1,1,1e1,1,2 e1,2,0e1,2,1e1,2,2a1,0,0a1,0,1a1,0,2a1,1 ,0a1,1,1a1,1,2a1,2,0a1,2,1a1,2,2 Alignment scores: H x W Attention : H x W c2 XFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention22 CNNF eatures: H x W x Dh0c1y0h1[START]y1h2y2c2y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0personpersonwearingz0,1z0,2z1,0z 1,1z1,2z2,0z2,1z2,2 Each timestep of decoder uses a different context vector that looks at different parts of the input imageDecoder: yt = gV(yt-1, ht-1, ct)New context vector at every time stepFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention23 CNNF eatures: H x W x Dh0c1y0h1[START]y1h2y2c2y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0h3y3c3y2personwearingpersonweari nghatz0,1z0,2z1,0z1,1z1,2z2,0z2,1z2,2 Each timestep of decoder uses a different context vector that looks at different parts of the input imageDecoder: yt = gV(yt-1, ht-1, ct)New context vector at every time stepFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention24 CNNF eatures: H x W x Dh0c1y0h1[START]y1h2y2c2y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0h3y3c3y2personwearinghath4y4c4y3 personwearinghat[END]z0,1z0,2z1,0z1,1z1, 2z2,0z2,1z2,2 Each timestep of decoder uses a different context vector that looks at different parts of the input imageDecoder.

8 Yt = gV(yt-1, ht-1, ct)New context vector at every time stepFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Extract spatial features from a pretrained CNNI mage Captioning with RNNs & Attention25 CNNF eatures: H x W x Dh0c1y0h1[START]y1h2y2c2y1Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015z0,0h3y3c3y2personwearinghath4y4c4y3 personwearinghat[END]z0,1z0,2z1,0z1,1z1, 2z2,0z2,1z2,2 This entire process is chooses its own Attention weights. No Attention supervision is requiredXu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015e1,0,0e1,0,1e1,0,2e1,1,0e1,1,1e1,1,2 e1,2,0e1,2,1e1,2,2a1,0,0a1,0,1a1,0,2a1,1 ,0a1,1,1a1,1,2a1,2,0a1,2,1a1,2,2 Alignment scores: H x W Attention : H x W XFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 202126 Soft attentionImage Captioning with AttentionXu et al, Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015 Figure copyright Kelvin Xu, Jimmy Lei Ba, Jamie Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S.

9 Zemel, and Yoshua Benchio, 2015. Reproduced with Attention (requires reinforcement learning)Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 202127 Image Captioning with AttentionXu et al, Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention , ICML 2015 Figure copyright Kelvin Xu, Jimmy Lei Ba, Jamie Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Benchio, 2015. Reproduced with Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Burns et al. Women also Snowboard: Overcoming Bias in Captioning Models ECCV 2018 Figures from Burns et al, copyright 2018. Reproduced with can detect Gender BiasAll images are CC0 Public domain: dog,Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Similar tasks in NLP - Language translation example29 Input: Sequence x = x1, x2,.., xTOutput: Sequence y = y1, y2,.., yTx0x1x2x3personneportantunchapeauFei-Fe i Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Similar tasks in NLP - Language translation exampleEncoder: h0= fW(z)where zt = RNN(xt, ut-1) fW(.)

10 Is MLPu is the hidden RNN state30 Input: Sequence x = x1, x2,.., xTOutput: Sequence y = y1, y2,.., yTx0z0z1x1z2x2z3x3h0personneportantuncha peauFei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Similar tasks in NLP - Language translation exampleEncoder: h0= fW(z)where zt = RNN(xt, ut-1) fW(.) is MLPu is the hidden RNN state31 Input: Sequence x = x1, x2,.., xTOutput: Sequence y = y1, y2,.., yTDecoder: yt = gV(yt-1, ht-1, c)where context vector c is often c = h0x0z0z1x1z2x2z3x3h0[START]y0h1[START]y1 h2y2y1h3y3y2personwearinghath4y4y3person wearinghat[END]cpersonneportantunchapeau Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Attention in NLP - Language translation example32x0z0z1x1z2x2z3x3h0e0e1e2e3 Compute alignments scores (scalars):fatt(.) is an MLPpersonneportantunchapeauBahdanau et al, Neural machine translation by jointly learning to align and translate , ICLR 2015 Fei-Fei Li, Ranjay Krishna, Danfei XuLecture 11 -May 06, 2021 Attention in NLP - Language translation example33x0z0z1x1z2x2z3x3h0e0e1e2e3a0a1a 2a3 Normalize to get Attention weights:0 < at, i, j < 1, Attention values sum to 1 Compute alignments scores (scalars):fatt(.)