Hierarchical Attention Networks for Document Classification

1 Proceedings of NAACL-HLT 2016, pages 1480 1489,San Diego, California, June 12-17, 2016 Association for Computational LinguisticsHierarchical Attention Networks for Document ClassificationZichao Yang1, Diyi Yang1, Chris Dyer1, Xiaodong He2, Alex Smola1, Eduard Hovy11 Carnegie Mellon University,2 Microsoft Research, Redmondfzichaoy, diyiy, cdyer, propose a Hierarchical Attention networkfor Document Classification . Our model hastwo distinctive characteristics: (i) it has a hier-archical structure that mirrors the hierarchicalstructure of documents; (ii) it has two levelsof Attention mechanisms applied at the word-and sentence-level, enabling it to attend dif-ferentially to more and less important con-tent when constructing the Document repre-sentation. Experiments conducted on six largescale text Classification tasks demonstrate thatthe proposed architecture outperform previousmethods by a substantial margin. Visualiza-tion of the Attention layers illustrates that themodel selects qualitatively informative wordsand IntroductionText Classification is one of the fundamental task inNatural Language Processing.

2 The goal is to as-sign labels to text. It has broad applications includ-ing topic labeling (Wang and Manning, 2012), senti-ment Classification (Maas et al., 2011; Pang and Lee,2008), and spam detection (Sahami et al., 1998).Traditional approaches of text Classification repre-sent documents with sparse lexical features, suchasn-grams, and then use a linear model or kernelmethods on this representation (Wang and Manning,2012; Joachims, 1998). More recent approachesused deep learning, such as convolutional neural net-works (Blunsom et al., 2014) and recurrent neuralnetworks based on long short-term memory (LSTM)(Hochreiter and Schmidhuber, 1997) to learn belly = don t evenlike scallops, andthese tasty time I in Phoenix, I willgo back 1:A simple example review from Yelp 2013 that con-sists of five sentences, delimited by period, question mark. Thefirst and third sentence delivers stronger meaning and inside,the worddelicious, a-m-a-z-i-n-gcontributes the most in defin-ing sentiment of the two neural-network based approaches totext Classification have been quite effective (Kim,2014; Zhang et al.)

3 , 2015; Johnson and Zhang, 2014;Tang et al., 2015), in this paper we test the hypoth-esis that better representations can be obtained byincorporating knowledge of Document structure inthe model architecture. The intuition underlying ourmodel is that not all parts of a Document are equallyrelevant for answering a query and that determiningthe relevant sections involves modeling the interac-tions of the words , not just their presence in primary contribution is a new neural archi-tecture (x2), the Hierarchical Attention Network(HAN) that is designed to capture two basic insightsabout Document structure. First, since documentshave a Hierarchical structure ( words form sentences,sentences form a Document ), we likewise construct adocument representation by first building represen-tations of sentences and then aggregating those intoa Document representation. Second, it is observedthat different words and sentences in a documentsare differentially informative. Moreover, the impor-1480tance of words and sentences are highly context de-pendent, the same word or sentence may be dif-ferentially important in different context ( ).

4 Toinclude sensitivity to this fact, our model includestwo levels of Attention mechanisms (Bahdanau et al.,2014; Xu et al., 2015) one at the word level andone at the sentence level that let the model topay more or less Attention to individual words andsentences when constructing the representation ofthe Document . To illustrate, consider the examplein Fig. 1, which is a short Yelp review where thetask is to predict the rating on a scale from 1 5. In-tuitively, the first and third sentence have strongerinformation in assisting the prediction of the rat-ing; within these sentences, the worddelicious,a-m-a-z-i-n-gcontributes more in implyingthe positive attitude contained in this review. At-tention serves two benefits: not only does it oftenresult in better performance, but it also provides in-sight into which words and sentences contribute tothe Classification decision which can be of value inapplications and analysis (Shen et al., 2014; Gao etal., 2014).

5 The key difference to previous work is that oursystem usescontextto discoverwhena sequence oftokens is relevant rather than simply filtering for (se-quences of) tokens, taken out of context. To evaluatethe performance of our model in comparison to othercommon Classification architectures, we look at sixdata sets (x3). Our model outperforms previous ap-proaches by a significant Hierarchical Attention NetworksThe overall architecture of the Hierarchical Atten-tion Network (HAN) is shown in Fig. 2. It con-sists of several parts: a word sequence encoder, aword-level Attention layer, a sentence encoder and asentence-level Attention layer. We describe the de-tails of different components in the following GRU-based sequence encoderThe GRU (Bahdanau et al., 2014) uses a gatingmechanism to track the state of sequences withoutusing separate memory cells. There are two types ofgates: the reset gatertand the update gatezt. Theytogether control how information is updated to the h21 h21 h22 h22 !

6 H22 !h22 h2T h2T !h2T !h2 Tuwuww21w21w22w22w2Tw2 Twordencoderwordattention h1 h1 !h1 !h1 h2 h2 !h2 !h2 hL hL !hL !hLususs1s1s2s2sLsL L Lsentenceencodersentenceattentionvvsoftm ax 21 21 22 22 2T 2T !h21 !h21 1 1 2 2 Figure 2: Hierarchical Attention At timet, the GRU computes the new state asht= (1 zt) ht 1+zt ht.(1)This is a linear interpolation between the previousstateht 1and the current new state htcomputedwith new sequence information. The gateztdecideshow much past information is kept and how muchnew information is updated as:zt= (Wzxt+Uzht 1+bz),(2)wherextis the sequence vector at timet. The can-didate state htis computed in a way similar to a tra-ditional recurrent neural network (RNN): ht= tanh(Whxt+rt (Uhht 1) +bh),(3)Herertis the reset gate which controls how muchthe past state contributes to the candidate state. Ifrtis zero, then it forgets the previous state. The resetgate is updated as follows:rt= (Wrxt+Urht 1+br)(4) Hierarchical AttentionWe focus on Document -level Classification in thiswork.

7 Assume that a Document hasLsentences1481siand each sentence [1, T]represents the words in theith proposed model projects the raw Document intoa vector representation, on which we build a classi-fier to perform Document Classification . In the fol-lowing, we will present how we build the documentlevel vector progressively from word vectors by us-ing the Hierarchical EncoderGiven a sentence with wordswit, t2[0, T], we first embed the words to vectorsthrough an embedding matrixWe,xij= use a bidirectional GRU (Bahdanau et al., 2014)to get annotations of words by summarizing infor-mation from both directions for words , and thereforeincorporate the contextual information in the anno-tation. The bidirectional GRU contains the forwardGRU !fwhich reads the sentencesifromwi1towiTand a backward GRU fwhich reads fromwiTtowi1:xit=Wewit, t2[1, T], !hit= !GRU(xit), t2[1, T], hit= GRU(xit), t2[T,1].We obtain an annotation for a given wordwitbyconcatenating the forward hidden state !

8 Hitandbackward hidden state hit, ,hit= [ !hit, hit],which summarizes the information of the whole sen-tence centered that we directly use word embeddings. Fora more complete model we could use a GRU to getword vectors directly from characters, similarly to(Ling et al., 2015). We omitted this for AttentionNot all words contribute equallyto the representation of the sentence , we introduce Attention mechanism to extractsuch words that are important to the meaning of thesentence and aggregate the representation of thoseinformative words to form a sentence vector. Specif-ically,uit= tanh(Wwhit+bw)(5) it=exp(u ituw) texp(u ituw)(6)si= t ithit.(7)That is, we first feed the word annotationhitthrougha one-layer MLP to getuitas a hidden represen-tation ofhit, then we measure the importance ofthe word as the similarity ofuitwith a word levelcontext vectoruwand get a normalized importanceweight itthrough a softmax function. After that,we compute the sentence vectorsi(we abuse the no-tation here) as a weighted sum of the word annota-tions based on the weights.

9 The context vectoruwcan be seen as a high level representation of a fixedquery what is the informative word over the wordslike that used in memory Networks (Sukhbaatar etal., 2015; Kumar et al., 2015). The word contextvectoruwis randomly initialized and jointly learnedduring the training EncoderGiven the sentence vectorssi,we can get a Document vector in a similar way. Weuse a bidirectional GRU to encode the sentences: !hi= !GRU(si), i2[1, L], hi= GRU(si), t2[L,1].We concatenate !hiand hjto get an annotation ofsentencei, ,hi= [ !hi, hi].hisummarizes theneighbor sentences around sentenceibut still focuson AttentionTo reward sentences that areclues to correctly classify a Document , we again useattention mechanism and introduce a sentence levelcontext vectorusand use the vector to measure theimportance of the sentences. This yieldsui= tanh(Wshi+bs),(8) i=exp(u ius) iexp(u ius),(9)v= i ihi,(10)wherevis the Document vector that summarizesall the information of sentences in a , the sentence level context vector can berandomly initialized and jointly learned during thetraining Document ClassificationThe Document vectorvis a high level representationof the Document and can be used as features for doc-1482ument Classification :p=softmax(Wcv+bc).

10 (11)We use the negative log likelihood of the correct la-bels as training loss:L= dlogpdj,(12)wherejis the label of Data setsWe evaluate the effectiveness of our model on sixlarge scale Document Classification data sets. Thesedata sets can be categorized into two types of doc-ument Classification tasks: sentiment estimation andtopic Classification . The statistics of the data sets aresummarized in Table 1. We use 80% of the data fortraining, 10% for validation, and the remaining 10%for test, unless stated reviewsare obtained from the Yelp DatasetChallenge in 2013, 2014 and 2015 (Tang et al.,2015). There are five levels of ratings from 1 to5 (higher is better).IMDB reviewsare obtained from (Diao et al.,2014). The ratings range from 1 to answersare obtained from (Zhang et al.,2015). This is a topic Classification task with 10classes: Society & Culture, Science & Mathe-matics, Health, Education & Reference, Com-puters & Internet, Sports, Business & Finance,Entertainment & Music, Family & Relation-ships and Politics & Government.