Hierarchical Attention Networks for Document Classification

1 Proceedings of NAACL-HLT 2016, pages 1480 1489,San Diego, California, June 12-17, 2016 Association for Computational LinguisticsHierarchical Attention Networks for Document ClassificationZichao Yang1, Diyi Yang1, Chris Dyer1, Xiaodong He2, Alex Smola1, Eduard Hovy11 Carnegie Mellon University,2 Microsoft Research, Redmondfzichaoy, diyiy, cdyer, propose a Hierarchical Attention networkfor Document Classification . Our model hastwo distinctive characteristics: (i) it has a hier-archical structure that mirrors the hierarchicalstructure of documents; (ii) it has two levelsof Attention mechanisms applied at the word-and sentence-level, enabling it to attend dif-ferentially to more and less important con-tent when constructing the Document repre-sentation.

2 Experiments conducted on six largescale text Classification tasks demonstrate thatthe proposed architecture outperform previousmethods by a substantial margin. Visualiza-tion of the Attention layers illustrates that themodel selects qualitatively informative wordsand IntroductionText Classification is one of the fundamental task inNatural Language Processing. The goal is to as-sign labels to text. It has broad applications includ-ing topic labeling (Wang and Manning, 2012), senti-ment Classification (Maas et al., 2011; Pang and Lee,2008), and spam detection (Sahami et al.)

3 , 1998).Traditional approaches of text Classification repre-sent documents with sparse lexical features, suchasn-grams, and then use a linear model or kernelmethods on this representation (Wang and Manning,2012; Joachims, 1998). More recent approachesused deep learning, such as convolutional neural net-works (Blunsom et al., 2014) and recurrent neuralnetworks based on long short-term memory (LSTM)(Hochreiter and Schmidhuber, 1997) to learn belly = don t evenlike scallops, andthese tasty time I in Phoenix, I willgo back 1:A simple example review from Yelp 2013 that con-sists of five sentences, delimited by period, question mark.

4 Thefirst and third sentence delivers stronger meaning and inside,the worddelicious, a-m-a-z-i-n-gcontributes the most in defin-ing sentiment of the two neural- network based approaches totext Classification have been quite effective (Kim,2014; Zhang et al., 2015; Johnson and Zhang, 2014;Tang et al., 2015), in this paper we test the hypoth-esis that better representations can be obtained byincorporating knowledge of Document structure inthe model architecture. The intuition underlying ourmodel is that not all parts of a Document are equallyrelevant for answering a query and that determiningthe relevant sections involves modeling the interac-tions of the words, not just their presence in primary contribution is a new neural archi-tecture (x2), the Hierarchical Attention network (HAN) that is designed to capture two basic insightsabout Document structure.

5 First, since documentshave a Hierarchical structure (words form sentences,sentences form a Document ), we likewise construct adocument representation by first building represen-tations of sentences and then aggregating those intoa Document representation. Second, it is observedthat different words and sentences in a documentsare differentially informative. Moreover, the impor-1480tance of words and sentences are highly context de-pendent, the same word or sentence may be dif-ferentially important in different context ( ). Toinclude sensitivity to this fact, our model includestwo levels of Attention mechanisms (Bahdanau et al.)

6 ,2014; Xu et al., 2015) one at the word level andone at the sentence level that let the model topay more or less Attention to individual words andsentences when constructing the representation ofthe Document . To illustrate, consider the examplein Fig. 1, which is a short Yelp review where thetask is to predict the rating on a scale from 1 5. In-tuitively, the first and third sentence have strongerinformation in assisting the prediction of the rat-ing; within these sentences, the worddelicious,a-m-a-z-i-n-gcontributes more in implyingthe positive attitude contained in this review.

7 At-tention serves two benefits: not only does it oftenresult in better performance, but it also provides in-sight into which words and sentences contribute tothe Classification decision which can be of value inapplications and analysis (Shen et al., 2014; Gao etal., 2014).The key difference to previous work is that oursystem usescontextto discoverwhena sequence oftokens is relevant rather than simply filtering for (se-quences of) tokens, taken out of context. To evaluatethe performance of our model in comparison to othercommon Classification architectures, we look at sixdata sets (x3).

8 Our model outperforms previous ap-proaches by a significant Hierarchical Attention NetworksThe overall architecture of the Hierarchical Atten-tion network (HAN) is shown in Fig. 2. It con-sists of several parts: a word sequence encoder, aword-level Attention layer, a sentence encoder and asentence-level Attention layer. We describe the de-tails of different components in the following GRU-based sequence encoderThe GRU (Bahdanau et al., 2014) uses a gatingmechanism to track the state of sequences withoutusing separate memory cells. There are two types ofgates: the reset gatertand the update gatezt.

9 Theytogether control how information is updated to the h21 h21 h22 h22 !h22 !h22 h2T h2T !h2T !h2 Tuwuww21w21w22w22w2Tw2 Twordencoderwordattention h1 h1 !h1 !h1 h2 h2 !h2 !h2 hL hL !hL !hLususs1s1s2s2sLsL L Lsentenceencodersentenceattentionvvsoftm ax 21 21 22 22 2T 2T !h21 !h21 1 1 2 2 Figure 2: Hierarchical Attention At timet, the GRU computes the new state asht= (1 zt) ht 1+zt ht.(1)This is a linear interpolation between the previousstateht 1and the current new state htcomputedwith new sequence information. The gateztdecideshow much past information is kept and how muchnew information is updated as:zt= (Wzxt+Uzht 1+bz),(2)wherextis the sequence vector at timet.

10 The can-didate state htis computed in a way similar to a tra-ditional recurrent neural network (RNN): ht= tanh(Whxt+rt (Uhht 1) +bh),(3)Herertis the reset gate which controls how muchthe past state contributes to the candidate state. Ifrtis zero, then it forgets the previous state. The resetgate is updated as follows:rt= (Wrxt+Urht 1+br)(4) Hierarchical AttentionWe focus on Document -level Classification in thiswork. Assume that a Document hasLsentences1481siand each sentence [1, T]represents the words in theith proposed model projects the raw Document intoa vector representation, on which we build a classi-fier to perform Document Classification .