Recurrent Neural Network Based Language Model

1 Recurrent Neural Network Based Language modelTom a s Mikolov1,2, Martin Karafi at1, Luk a s Burget1, Jan Honza Cernock y1, Sanjeev Khudanpur21 Speech@FIT, Brno University of Technology, Czech Republic2 Department of Electrical and Computer Engineering, Johns Hopkins University, new Recurrent Neural Network Based Language Model (RNNLM) with applications to speech recognition is presented. Re-sults indicate that it is possible to obtain around 50% reductionof perplexity by using mixture of several RNN LMs, comparedto a state of the art backoff Language Model . Speech recognitionexperiments show around 18% reduction of word error rate onthe Wall Street Journal task when comparing models trained onthe same amount of data, and around 5% on the much harderNIST RT05 task, even when the backoff Model is trained onmuch more data than the RNN LM.

2 We provide ample empiri-cal evidence to suggest that connectionist Language models aresuperior to standard n-gram techniques, except their high com-putational (training) Terms: Language modeling, Recurrent Neural networks,speech recognition1. IntroductionSequential data prediction is considered by many as a key prob-lem in machine learning and artificial intelligence (see for ex-ample [1]). The goal of statistical Language modeling is topredict the next word in textual data given context; thus weare dealing with sequential data prediction problem when con-structing Language models. Still, many attempts to obtain suchstatistical models involve approaches that are very specific forlanguage domain - for example, assumption that natural lan-guage sentences can be described by parse trees, or that weneed to consider morphology of words, syntax and the most widely used and general models, Based on n-gram statistics, assume that Language consists of sequences ofatomic symbols - words - that form sentences, and where theend of sentence symbol plays important and very special is questionable if there has been any significant progressin Language modeling over simple n-gram models (see for ex-ample [2] for review of advanced techniques).

3 If we would mea-sure this progress by ability of models to better predict sequen-tial data, the answer would be that considerable improvementhas been achieved - namely by introduction of cache modelsand class- Based models. While many other techniques havebeen proposed, their effect is almost always similar to cachemodels (that describe long context information) or class-basedmodels (that improve parameter estimation for short contexts bysharing parameters between similar words).If we would measure success of advanced Language Model -ing techniques by their application in practice, we would haveto be much more skeptical. Language models for real-worldspeech recognition or machine translation systems are built onhuge amounts of data, and popular belief says that more datais all we need.

4 Models coming from research tend to be com-INPUT(t)OUTPUT(t)CONTEXT(t)CONTEXT(t -1)Figure 1:Simple Recurrent Neural and often work well only for systems Based on very limitedamounts of training data. In fact, most of the proposed advancedlanguage modeling techniques provide only tiny improvementsover simple baselines, and are rarely used in Model descriptionWe have decided to investigate Recurrent Neural networks formodeling sequential data. Using artificial Neural networks instatistical Language modeling has been already proposed byBengio [3], who used feedforward Neural networks with fixed-length context. This approach was exceptionally successfuland further investigation by Goodman [2] shows that this sin-gle Model performs better than mixture of several other modelsbased on other techniques, including class- Based Model .

5 Later,Schwenk [4] has shown that Neural Network Based models pro-vide significant improvements in speech recognition for severaltasks against good baseline major deficiency of Bengio s approach is that a feedfor-ward Network has to use fixed length context that needs to bespecifiedad hocbefore training. Usually this means that neuralnetworks see only five to ten preceding words when predictingthe next one. It is well known that humans can exploit longercontext with great success. Also, cache models provide comple-mentary information to Neural Network models, so it is naturalto think about a Model that would encode temporal informationimplicitly for contexts with arbitrary Neural networks do not use limited size of con-text.

6 By using Recurrent connections, information can cycle in-Copyright 2010 ISCA26-30 September 2010, Makuhari, Chiba, JapanINTERSPEECH 20101045side these networks for arbitrarily long time (see [5]). However,it is also often claimed that learning long-term dependencies bystochastic gradient descent can be quite difficult [6].In our work, we have used an architecture that is usuallycalled asimple Recurrent Neural networkor Elman Network [7].This is probably the simplest possible version of Recurrent neu-ral Network , and very easy to implement and train. The networkhas an input layerx, hidden layers(also called context layeror state) and output layery. Input to the Network in timetisx(t), output is denoted asy(t), ands(t)is state of the Network (hidden layer).

7 Input vectorx(t)is formed by concatenatingvectorwrepresenting current word, and output from neurons incontext layersat timet 1. Input, hidden and output layersare then computed as follows:x(t) =w(t) +s(t 1)(1)sj(t) =f Xixi(t)uji!(2)yk(t) =g Xjsj(t)vkj!(3)wheref(z)is sigmoid activation function:f(z) =11 +e z(4)andg(z)is softmax function:g(zm) =ezmPkezk(5)For initialization,s(0)can be set to vector of small values, when processing a large amount of data, initialization isnot crucial. In the next time steps,s(t+1)is a copy ofs(t). In-put vectorx(t)represents word in timetencoded using1-of-Ncoding and previous context layer - size of vectorxis equal tosize of vocabularyV(this can be in practice30 000 200 000)plus size of context layer.

8 Size of context (hidden) layersisusually30 500hidden units. Based on our experiments, sizeof hidden layer should reflect amount of training data - for largeamounts of data, large hidden layer is are trained in several epochs, in which all datafrom training corpus are sequentially presented. Weights areinitialized to small values (random Gaussian noise with zeromean ). To train the Network , we use the stan-dard backpropagation algorithm with stochastic gradient de-scent. Starting learning rate is = After each epoch,the Network is tested on validation data. If log-likelihood ofvalidation data increases, training continues in new epoch. If nosignificant improvement is observed, learning rate is halvedat start of each new epoch.

9 After there is again no signifi-cant improvement, training is finished. Convergence is usuallyachieved after 10-20 our experiments, networks do not overtrain significantly,even if very large hidden layers are used - regularization of net-works to penalize large weights did not provide any significantimprovements. Output layery(t)represents probability dis-tribution of next word given previous wordw(t)and context1 Consequently, time needed to train optimal Network increases fasterthan just linearly with increased amount of training data: vocabularygrowth increases the input and output layer sizes, and also the optimalhidden layer size increases with more training (t 1). Softmax ensures that this probability distribution isvalid, (t)>0for any wordmandPkyk(t) = each training step, error vector is computed according tocross entropy criterion and weights are updated with the stan-dard backpropagation algorithm:error(t) = desired(t) y(t)(6)wheredesiredis a vector using1-of-Ncoding representing theword that should have been predicted in a particular context andy(t)is the actual output from the that training phase and testing phase in statistical lan-guage modeling usually differs in the fact that models do not getupdated as test data are being processed.

10 So, if a new person-name occurs repeatedly in the test set, it will repeatedly get avery small probability, even if it is composed of known can be assumed that such long term memory should not re-side in activation of context units (as these change very rapidly),but rather in synapses themselves - that the Network should con-tinue training even during testing phase. We refer to such modelasdynamic. For dynamic Model , we use fixed learning rate = While in training phase all data are presented to net-work several times in epochs, dynamic Model gets updated justonce as it processes testing data. This is of course not optimalsolution, but as we shall see, it is enough to obtain large perplex-ity reductions against static models.