Word2Vec Tutorial Part II: The Continuous Bag-of …

1 By Alex Minnaaron Mon 18 May 2015 Category: deep LearningLatent Dirichlet Allocation inScala part II - The Code Word2 Vec Tutorial part II: TheContinuous Bag-of -Words ModelIn the previous post the concept of word vectors was explained as was thederivation of the skip-gram model. In this post we will explore the other Word2 Vecmodel - the Continuous Bag-of -words (CBOW) model. If you understand the skip-gram model then the CBOW model should be quite straight-forward because inmany ways they are mirror images of each other. For instance, if you look at themodel diagramit looks like the skip-gram model with the inputs and outputs reversed.

2 The inputlayer consists of the one-hot encoded input context words for a wordwindow of size and vocabulary of size . The hidden layer is an N-dimensionalvector . Finally, the output layer is output word in the training example which isalso one-hot encoded. The one-hot encoded input vectors are connected to thehidden layer via a weight matrix and the hidden layer is connected to{,..,}x1xCCVhyV NWthe output layer via a wieght matrix .Forward PropagationWe must first understand how the output is computed from the input ( forwardpropagation). The following assumes that we know the input and output weightmatrices (I will explain how these are actually learned in the next section).

3 The firststep is to evaluate the output of the hidden layer . This is computed bywhich is the average of the input vectors weighted by the matrix . It is worthnoting that this hidden layer output computation is one of the only differencesbetween the Continuous Bag-of -words model and the skip-gram model (in terms ofthem being mirror images of course). Next we compute the inputs to each node inthe output layerwhere is the column of the output matrix . And finally we compute theoutput of the output layer. The output is obtained by passing the input throught the soft-max that we know how forward propagation works we can learn the weight matrices and.

4 learning the Weight Matrices withBackpropagationIn the process of learning the wieght matrices and , we begin with randomlyinitialized values. We then sequentially feed training examples into our model andobserve the error which is some function of the difference between the expectedoutput and the actual output. We then compute the gradient of this error withrespect to the elements of both weight matrices and correct them in the direction ofthis gradient. This general optimization procedure is known as stochastic gradientdescent (or sgd) but the method by which the gradients are derived is known VW hh=W ()1C i=1 CxiW= hujv wjTv wjjthW yjuj=p(|.)

5 ,)=yjwyjw1wCexp()ujexp() V=1j u jWW WW first step is to define the loss function. The objective is to maximize theconditional probability of the output word given the input context, therefore ourloss function will beWhere is the index of the the actual output word. The next step is to derive theupdate equation for the hidden-output layer weights , then derive the weightsfor the input-hidden layer weights Updating the hidden-output layer weightsThe first step is to compute the derivative of the loss function with respect to theinput to the node in the output layer.

6 Where if otherwise . This is simply the prediction error of node in the output layer. Next we take the derivative of with respect to the outputweight using the chain that we have the gradient with respect to an arbitrary output weight , wecan define the stochastic gradient descent logp(|)wOwI= logexp()uj =1j Vuj = h logexp( h)vwOT =1j Vvwj T(1)(2)(3)j W WEjthuj= E ujyjtj=1tjj=j =0tjjEw ij E w ij= E uj uj w ij=( ) yjtjhi(4)(5)w ij= ( ) w(new) ijw(old) ijyjtjhi= ( ) hv wj(new)v wj(old)yjtjwhere is the learning the input-hidden layer weightsNow let's try to derive a similar update equation for the input weights.

7 The firststep is to compute the derivative of with respect to an arbitrary hidden node (again using the chain rule).where the sum is do to the fact that the hidden layer node is connected to eachnode of the output layer and therefore each prediction error must be next step is to compute the derivative of with respect to an arbitrary inputweight .Where is an N-dimensional vector of elements from . However, since the inputs are one-hot encoded, only one row ofthe matrix will be nonzero. Thus the final stochastic gradientdescent equation for the input weights iswhere is the word in the input Tutorial part I: The Skip-Gram ModelDistributed Representations of Words and Phrases and their Compositionality, Mikolovet Language Processing (almost) from Scratch, Collobert et al.

8 >0wijEhi E hi= j=1V E uj uj hi=( ) j=1 Vyjtjw ij(6)(7)hiEwki E wki= E hi hi wki=( ) j=1 Vyjtjw ij1 Cxk=(x EH)1C(8)(9)(10)EH( ) Vj=1yjtjw iji=1,..,NxN V(x EH)1C= EHv wI,c(new)v wI,c(old)1 CwI,ccthProudly powered by Pelican, which takes great advantage of Python. LaunchYardNatural Language Processing (almost) from Scratch, Collobert et al. &RPPHQWVDOH[PLQQDDUGRWFRP /RJLQ 6 KDUH 6 RUW E\ %HVW6 WDUW WKH GLVFXVVLRQ %H WKH ILUVW WR FRPPHQW 6 XEVFULEH $GG 'LVTXV WR \RXU VLWH $GG 'LVTXV $GGd3 ULYDF\ 5 HFRPPHQG]