Hierarchical Attention Networks for Document Classification

1 Proceedings of NAACL-HLT 2016, pages 1480 1489,San Diego, California, June 12-17, 2016 Association for Computational LinguisticsHierarchical Attention Networks for Document ClassificationZichao Yang1, Diyi Yang1, Chris Dyer1, Xiaodong He2, Alex Smola1, Eduard Hovy11 Carnegie Mellon University,2 Microsoft Research, Redmondfzichaoy, diyiy, cdyer, propose a Hierarchical Attention networkfor Document Classification . Our model hastwo distinctive characteristics: (i) it has a hier-archical structure that mirrors the hierarchicalstructure of documents; (ii) it has two levelsof Attention mechanisms applied at the word-and sentence-level, enabling it to attend dif-ferentially to more and less important con-tent when constructing the Document repre-sentation. Experiments conducted on six largescale text Classification tasks demonstrate thatthe proposed architecture outperform previousmethods by a substantial margin.

2 Visualiza-tion of the Attention layers illustrates that themodel selects qualitatively informative wordsand IntroductionText Classification is one of the fundamental task inNatural Language Processing. The goal is to as-sign labels to text. It has broad applications includ-ing topic labeling (Wang and Manning, 2012), senti-ment Classification (Maas et al., 2011; Pang and Lee,2008), and spam detection (Sahami et al., 1998).Traditional approaches of text Classification repre-sent documents with sparse lexical features, suchasn-grams, and then use a linear model or kernelmethods on this representation (Wang and Manning,2012; Joachims, 1998). More recent approachesused deep learning, such as convolutional neural net-works (Blunsom et al., 2014) and recurrent neuralnetworks based on long short-term memory (LSTM)(Hochreiter and Schmidhuber, 1997) to learn belly = don t evenlike scallops, andthese tasty time I in Phoenix, I willgo back 1:A simple example review from Yelp 2013 that con-sists of five sentences, delimited by period, question mark.

3 Thefirst and third sentence delivers stronger meaning and inside,the worddelicious, a-m-a-z-i-n-gcontributes the most in defin-ing sentiment of the two neural- network based approaches totext Classification have been quite effective (Kim,2014; Zhang et al., 2015; Johnson and Zhang, 2014;Tang et al., 2015), in this paper we test the hypoth-esis that better representations can be obtained byincorporating knowledge of Document structure inthe model architecture. The intuition underlying ourmodel is that not all parts of a Document are equallyrelevant for answering a query and that determiningthe relevant sections involves modeling the interac-tions of the words, not just their presence in primary contribution is a new neural archi-tecture (x2), the Hierarchical Attention network (HAN) that is designed to capture two basic insightsabout Document structure. First, since documentshave a Hierarchical structure (words form sentences,sentences form a Document ), we likewise construct adocument representation by first building represen-tations of sentences and then aggregating those intoa Document representation.

4 Second, it is observedthat different words and sentences in a documentsare differentially informative. Moreover, the impor-1480tance of words and sentences are highly context de-pendent, the same word or sentence may be dif-ferentially important in different context ( ). Toinclude sensitivity to this fact, our model includestwo levels of Attention mechanisms (Bahdanau et al.,2014; Xu et al., 2015) one at the word level andone at the sentence level that let the model topay more or less Attention to individual words andsentences when constructing the representation ofthe Document . To illustrate, consider the examplein Fig. 1, which is a short Yelp review where thetask is to predict the rating on a scale from 1 5. In-tuitively, the first and third sentence have strongerinformation in assisting the prediction of the rat-ing; within these sentences, the worddelicious,a-m-a-z-i-n-gcontributes more in implyingthe positive attitude contained in this review.

5 At-tention serves two benefits: not only does it oftenresult in better performance, but it also provides in-sight into which words and sentences contribute tothe Classification decision which can be of value inapplications and analysis (Shen et al., 2014; Gao etal., 2014).The key difference to previous work is that oursystem usescontextto discoverwhena sequence oftokens is relevant rather than simply filtering for (se-quences of) tokens, taken out of context. To evaluatethe performance of our model in comparison to othercommon Classification architectures, we look at sixdata sets (x3). Our model outperforms previous ap-proaches by a significant Hierarchical Attention NetworksThe overall architecture of the Hierarchical Atten-tion network (HAN) is shown in Fig. 2. It con-sists of several parts: a word sequence encoder, aword-level Attention layer, a sentence encoder and asentence-level Attention layer.

6 We describe the de-tails of different components in the following GRU-based sequence encoderThe GRU (Bahdanau et al., 2014) uses a gatingmechanism to track the state of sequences withoutusing separate memory cells. There are two types ofgates: the reset gatertand the update gatezt. Theytogether control how information is updated to the h21 h21 h22 h22 !h22 !h22 h2T h2T !h2T !h2 Tuwuww21w21w22w22w2Tw2 Twordencoderwordattention h1 h1 !h1 !h1 h2 h2 !h2 !h2 hL hL !hL !hLususs1s1s2s2sLsL L Lsentenceencodersentenceattentionvvsoftm ax 21 21 22 22 2T 2T !h21 !h21 1 1 2 2 Figure 2: Hierarchical Attention At timet, the GRU computes the new state asht= (1 zt) ht 1+zt ht.(1)This is a linear interpolation between the previousstateht 1and the current new state htcomputedwith new sequence information. The gateztdecideshow much past information is kept and how muchnew information is updated as:zt= (Wzxt+Uzht 1+bz),(2)wherextis the sequence vector at timet.

7 The can-didate state htis computed in a way similar to a tra-ditional recurrent neural network (RNN): ht= tanh(Whxt+rt (Uhht 1) +bh),(3)Herertis the reset gate which controls how muchthe past state contributes to the candidate state. Ifrtis zero, then it forgets the previous state. The resetgate is updated as follows:rt= (Wrxt+Urht 1+br)(4) Hierarchical AttentionWe focus on Document -level Classification in thiswork. Assume that a Document hasLsentences1481siand each sentence [1, T]represents the words in theith proposed model projects the raw Document intoa vector representation, on which we build a classi-fier to perform Document Classification . In the fol-lowing, we will present how we build the documentlevel vector progressively from word vectors by us-ing the Hierarchical EncoderGiven a sentence with wordswit, t2[0, T], we first embed the words to vectorsthrough an embedding matrixWe,xij= use a bidirectional GRU (Bahdanau et al.)

8 , 2014)to get annotations of words by summarizing infor-mation from both directions for words, and thereforeincorporate the contextual information in the anno-tation. The bidirectional GRU contains the forwardGRU !fwhich reads the sentencesifromwi1towiTand a backward GRU fwhich reads fromwiTtowi1:xit=Wewit, t2[1, T], !hit= !GRU(xit), t2[1, T], hit= GRU(xit), t2[T,1].We obtain an annotation for a given wordwitbyconcatenating the forward hidden state !hitandbackward hidden state hit, ,hit= [ !hit, hit],which summarizes the information of the whole sen-tence centered that we directly use word embeddings. Fora more complete model we could use a GRU to getword vectors directly from characters, similarly to(Ling et al., 2015). We omitted this for AttentionNot all words contribute equallyto the representation of the sentence , we introduce Attention mechanism to extractsuch words that are important to the meaning of thesentence and aggregate the representation of thoseinformative words to form a sentence vector.

9 Specif-ically,uit= tanh(Wwhit+bw)(5) it=exp(u ituw) texp(u ituw)(6)si= t ithit.(7)That is, we first feed the word annotationhitthrougha one-layer MLP to getuitas a hidden represen-tation ofhit, then we measure the importance ofthe word as the similarity ofuitwith a word levelcontext vectoruwand get a normalized importanceweight itthrough a softmax function. After that,we compute the sentence vectorsi(we abuse the no-tation here) as a weighted sum of the word annota-tions based on the weights. The context vectoruwcan be seen as a high level representation of a fixedquery what is the informative word over the wordslike that used in memory Networks (Sukhbaatar etal., 2015; Kumar et al., 2015). The word contextvectoruwis randomly initialized and jointly learnedduring the training EncoderGiven the sentence vectorssi,we can get a Document vector in a similar way. Weuse a bidirectional GRU to encode the sentences: !

10 Hi= !GRU(si), i2[1, L], hi= GRU(si), t2[L,1].We concatenate !hiand hjto get an annotation ofsentencei, ,hi= [ !hi, hi].hisummarizes theneighbor sentences around sentenceibut still focuson AttentionTo reward sentences that areclues to correctly classify a Document , we again useattention mechanism and introduce a sentence levelcontext vectorusand use the vector to measure theimportance of the sentences. This yieldsui= tanh(Wshi+bs),(8) i=exp(u ius) iexp(u ius),(9)v= i ihi,(10)wherevis the Document vector that summarizesall the information of sentences in a , the sentence level context vector can berandomly initialized and jointly learned during thetraining Document ClassificationThe Document vectorvis a high level representationof the Document and can be used as features for doc-1482ument Classification :p=softmax(Wcv+bc).(11)We use the negative log likelihood of the correct la-bels as training loss:L= dlogpdj,(12)wherejis the label of Data setsWe evaluate the effectiveness of our model on sixlarge scale Document Classification data sets.