Flipped Classroom Questions on Recurrent …

1 Flipped Classroom Questions on Recurrent Networks and AttentionMichael CollinsQuestion 1:Consider the equations for a Recurrent model mapping an input source language an output target language :Inputs:A eachxj : i,j,n Ris a parameter for aligning source wordito target wordjgiven source sentence lengthnComputational Graph: Encoding step:use a bi-directional LSTM to map the input a new sequenceu(1)..u(n). Decoding:Forj= Defineai,j= COMPLETE CODE HERE Setc(j)= ni=1ai,ju(i) (j)=g(Whcc(j)+bh) l(j)=V (j)+ yj= arg maxyl(j)y j=j+ 1 Untilyj 1=STOP 1 Question 1a:How would you complete the code above to give similar behavior to IBM model 2?

2 Recallthat in IBM model 2 we have alignment parametersq(i|j,m,n)and translation parameterst(f|e).Question 1b:Draw the computational graph for this model (you can take the inputs to this computationgraph to beu(1)..u(n); there is no need to draw the computational graph for the encoding 1c:Suppose we have a training set consisting ofthe dog/le chien the cat/le chatandthehospital/l h opital. How would you set the i,j,n,mparameters to model this data?1 Question 1d:Suppose we now wish for each vector of alignment variablesaj= a1, ,j todepend on the previous alignment variablesaj 1, and the previous target-language wordyj 1.)

3 How wouldyou alter the equations in the figure to achieve this?Question 2:Consider the attention-based model for translation given in the lecture slides. The followingpseudo-code can be used to calculate the distributionp(y| 1, )for any input target language 1:Inputs:Source language , target language 1 Goal:Compute the conditional distributionp(y| 1, ) Encoding step:use an LSTM to (1)..u(n) Decoding step:Forj= (k 1) Fori= ,si,j=A( (j 1),u(i); A) Fori= ,ai,j=exp{si,j} ni=1exp{si,j} Setc(j)= ni=1ai,ju(i) (j)=LSTM(CONCAT(yj 1,c(j)), (j 1); D) l(j)=V CONCAT( (j),yj 1,c(j)) + , q(j)=LS(l(j)), Output:q(j)such thatq(j)y= logp(y| 1, )(Continued over page.)

4 2 Consider a beam-search algorithm for decoding with this model. Assume for simplicity that the input tothe algorithm is an integermspecifying the length of the target language sentence. You can make use of thefollowing primitives: BEAM(k)fork= a set ofitems. Eachitemis a pair( ,score) apartial translation, andscore=k j=1logp(yj| 1, ) The function ADD( ,score)adds a partial translation together with a score to BEAM(k). Ifthe partial translation is not in the top10highest scoring partial translations in BEAM(k), it is notadded. If after the addition there are more than 10 items in the beam, only the top 10 highest scoringitems are retained in the beam.

5 (The beam keeps the top 10 most likely translations at each point) INIT(BEAM)initializes BEAM(k)to be the empty set fork 1, and initializes BEAM(0)to containa single entry with partial translation equal to (the empty string), and score equal to0. foreach( ,score) BEAM(k) initializes aforeachloop over the items in BEAM(k) We can use the computational graph in the pseudo-code above to calculate the distributionlogp(y| 1, )for any 1 Finally, ARGMAX(BEAM)returns the highest ,score pair in BEAM(1) BEAM(2).. BEAM(m)such thatyk=STOPQ uestion 2a:Write pseudo-code for a beam-search algorithm using the above primitives.

6 The algorithmshould take a source language , and an integermspecifying the maximum output length,as 2b:If we naively calculatep(y| 1, )using the code given above, there will be a lot of repeated (wasted) computation. How would you make thealgorithm more efficient, by caching some computation?3 Question 3:Consider the function that maps a vectors Rdto a new vectora [0,1]dthrough thesoftmax function:aj=exp{sj} di=1exp{si}This is the function frequently used in attention-based eachj= , what is the value for aj sjFor eachj,j such thatj6=j , what is the value for aj sj Hint: recall the quotient rule for differentiation.

7 Iff(x) =g(x)/h(x), andf (x)/g (x)/h (x)is thederivative off(x)/g(x)/h(x)respectively, thenf (x) =g (x)h(x) g(x)h (x)(h(x))24