Hierarchical Attention Networks for Document Classification

Proceedings of NAACL-HLT 2016, pages 1480 1489,San Diego, California, June 12-17, 2016 Association for Computational LinguisticsHierarchical Attention Networks for Document ClassificationZichao Yang1, Diyi Yang1, Chris Dyer1, Xiaodong He2, Alex Smola1, Eduard Hovy11 Carnegie Mellon University,2 Microsoft Research, Redmondfzichaoy, diyiy, cdyer, propose a Hierarchical Attention networkfor Document Classification . Our model hastwo distinctive characteristics: (i) it has a hier-archical structure that mirrors the hierarchicalstructure of documents; (ii) it has two levelsof Attention mechanisms applied at the word-and sentence-level, enabling it to attend dif-ferentially to more and less important con-tent when constructing the Document repre-sentation. Experiments conducted on six largescale text Classification tasks demonstrate thatthe proposed architecture outperform previousmethods by a substantial margin. Visualiza-tion of the Attention layers illustrates that themodel selects qualitatively informative wordsand IntroductionText Classification is one of the fundamental task inNatural Language Processing.

the word as the similarity of u it with a word level context vector u w and get a normalized importance weight it through a softmax function. After that, wecomputethesentencevector si (weabusetheno-tation here) as a weighted sum of the word annota-tions based on the weights. The context vector u w can be seen as a high level representation of a xed

Loading..

Tags:

  Words

Information

Related search queries