Heterogeneous Graph Attention Networks for Semi …

1 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing, pages 4821 4830,Hong Kong, China, November 3 7, 2019 Association for Computational Linguistics4821 Heterogeneous Graph Attention Networks for Semi-supervisedShort Text ClassificationLinmei Hu1, Tianchi Yang1, Chuan Shi*1, Houye Ji1, Xiaoli Li21 Beijing University of Posts and Telecommunications, China2 Institute for Infocomm Research, text classification has found rich andcritical applications in news and tweet taggingto help users find relevant information. Due tolack of labeled training data in many practicaluse cases, there is a pressing need for study-ing semi-supervised short text existing studies focus on long texts andachieve unsatisfactory performance on shorttexts due to the sparsity and limited labeleddata.

2 In this paper, we propose a novel hetero-geneous Graph neural network based methodfor semi-supervised short text classification,leveraging full advantage of few labeled dataand large unlabeled data through informationpropagation along the Graph . In particular,we first present a flexible HIN (heterogeneousinformation network) framework for model-ing the short texts, which can integrate anytype of additional information as well as cap-ture their relations to address the semanticsparsity. Then, we proposeHeterogeneousGraphATtention Networks (HGAT) to embedthe HIN for short text classification based ona dual-level Attention mechanism, includingnode-level and type-level attentions. The at-tention mechanism can learn the importance ofdifferent neighboring nodes as well as the im-portance of different node (information) typesto a current node.

3 Extensive experimentalresults have demonstrated that our proposedmodel outperforms state-of-the-art methodsacross six benchmark datasets IntroductionWith the rapid development of online social me-dia and e-commerce, short texts, such as onlinenews, queries, reviews, tweets, are increasinglywidespread on the Internet (Song et al.,2014).Short text classification can be widely applied inmany domains, ranging from sentiment analysisto news tagging/categorization and query intentclassification (Aggarwal and Zhai,2012;Menget al.,2018). In many practical scenarios, thelabeled data is scarce, while human labeling istime-consuming and may require expert knowl-edge (Aggarwal and Zhai,2012). As a conse-quence, there is a pressing need for studying semi-supervised short text classification with a rela-tively small number of labeled training , semi-supervised short text clas-sification is nontrivial due to the following chal-lenges.

4 Firstly, short texts are usually seman-tically sparse and ambiguous, lacking contexts(Phan et al.,2008). While some methods havebeen proposed to incorporate additional informa-tion such as entities (Wang et al.,2013,2017),they are unable to consider the relational data suchas the semantic relations among entities. Sec-ondly, the labeled training data is limited, whichleads to traditional and neural supervised meth-ods (Wang and Manning,2012;Kim,2014;Zhanget al.,2015) ineffective. As such, how to make fulluse of the limited labeled data and large number ofunlabeled data has become a key problem for shorttext classification (Aggarwal and Zhai,2012). Fi-nally, we need to capture the importance of dif-ferent information that is incorporated to addresssparsity at multiple granularity levels and reducethe weights of noisy information to achieve moreaccurate classification this work, we propose a novelheterogeneousgraph neural network based methodfor semi-supervised short text classification, which makesfull use of both limited labeled data and large un- labeled data by allowing information propagationthrough our automatically constructed Graph .

5 Par-ticularly, we first present a flexible HIN frame-work for modeling the short texts, which is able toincorporate any additional information ( , enti-ties and topics) as well as capture the rich relationsamong the texts and the additional , we propose Heterogeneous Graph Attention4822networks (HGAT) to embed the HIN for short textclassification based on a new dual-level attentionmechanism including node-level and type-level at-tentions. Our HGAT method considers the hetero-geneity of different node types. Additionally, thedual-level Attention mechanism captures both theimportance of different neighboring nodes (reduc-ing the weights of noisy information) and the im-portance of different node (information) types to acurrent node. The main contributions of this papercan be summarized as follows.

6 1) To the best of our knowledge,this is the firstattemptto model short texts as well as additionalinformation with an HIN and adapt Graph neuralnetworks on the HIN for semi-supervised ) We propose novelheterogeneous Graph at-tention Networks (HGAT) for the HIN embeddingbased on a new dual-level Attention mechanismwhich can learn the importance of different neigh-boring nodes and the importance of different node(information) types to a current ) Extensive experimental results have demon-strated that our proposed HGAT model signifi-cantly outperforms seven state-of-the-art methodsacross six benchmark Related Traditional Text ClassificationTraditional text classification methods such asSVM (Drucker et al.,1999) need a feature en-gineering step for text representation.

7 The mostcommonly used features are BoW and TF-IDF(Blei et al.,2003). Some recent studies (Rousseauet al.,2015;Wang et al.,2016) model texts asgraphs and extract path based features for classi-fication. Despite its initial success on formal andwell-edited texts, all these methods fail to achievesatisfactory performance on short text classifica-tion, due to the insufficient features incurred byshort texts. To address the problem, efforts havebeen made to enrich the semantics ofshort example,Phan et al.(2008) extracted the la-tent topics of the short texts with the help of an ex-ternal et al.(2013) introduced exter-nal entity information from Knowledge Bases, , these methods are not able to achievegood performance as the feature engineering steprelies on domain Deep neural Networks for TextClassificationDeep neural Networks which automatically rep-resent texts as embeddings, have been widelyused for text classification.

8 Two representativedeep neural models, such as RNNs (Liu et al.,2016;Sinha et al.,2018) and CNNs, (Kim,2014;Shimura et al.,2018) have shown their power inmany NLP tasks, including text classification. Toadapt it toshort text classification, several meth-ods have been proposed. For example,Zhang et al.(2015) designs a character-level CNN which al-leviates the sparsity by mining different levels ofinformation within the et al.(2017)incorporates the entities and concepts from KBsto enrich the semantics of short texts. However,these methods cannot capture the semantic rela-tions ( , entity relations) and rely heavily on thenumber of training data. Clearly, lacking of train-ing data is still a key bottleneck that prohibits themfrom successful practical Semi-supervised Text ClassificationConsidering the cost of human labeling and thefact that unlabeled texts also provide valuable in-formation, semi-supervised methods have peenproposed.

9 They can be categorized into twoclasses: (1) latent variable models (Lu and Zhai,2008;Chen et al.,2015); and (2) embedding-basedmodels (Meng et al.,2018). The former mainlyextend topic model by user-provided seed infor-mation and then infer the documents labels basedon posterior category-topic assignment. The lat-ter use seed information to derive embeddingsfor documents and label names for text classifica-tion. For example, PTE (Tang et al.,2015) mod-els the documents, words and labels with graphsand learns text (node) embeddings for et al.(2018) leveraged seed in-formation to generate pseudo- labeled documentsfor et al.(2015) used a semi-supervised learning method based on SVM to la-bel the unlabeled documents in an iterative , Graph convolutional Networks (GCN)have received wide Attention for semi-supervisedclassification (Kipf and Welling,2017).

10 TextGCN(Yao et al.,2019) models the whole text corpusas a document -word Graph and applies GCN forclassification. However, all these methods focuson long texts. In addition, they fail to use attentionmechanisms to capture important Our Proposed MethodIn this paper, we propose a novel heteroge-neous Graph neural network based method forsemi-supervised short text classification, whichtakes full advantage of both limited labeled dataand large unlabeled data by allowing informationpropagation along the Graph . Our method includestwo steps. Particularly, to alleviate the sparsity ofshort texts, we first present a flexible HIN frame-work for modeling the short texts, which can in-corporate any additional information as well ascapture the rich relations among the short textsand the added information.