Attention-based LSTM for Aspect-level Sentiment Classification

1 Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 606 615,Austin, Texas, November 1-5, 2016 Association for Computational LinguisticsAttention-based LSTM for Aspect-level Sentiment ClassificationYequan WangandMinlie HuangandLi Zhao*andXiaoyan ZhuState Key Laboratory on Intelligent Technology and SystemsTsinghua National Laboratory for Information Science and TechnologyDepartment of Computer Science and Technology, Tsinghua University, Beijing 100084, China*Microsoft Research Sentiment Classification is a fine-grained task in Sentiment analysis. Since itprovides more complete and in-depth results, Aspect-level Sentiment analysis has receivedmuch attention these years. In this paper, wereveal that the Sentiment polarity of a sentenceis not only determined by the content but isalso highly related to the concerned instance, The appetizers are ok, but theservice is slow.

2 , for aspecttaste, the polar-ity is positive while forservice, the polarityis negative. Therefore, it is worthwhile to ex-plore the connection between an aspect andthe content of a sentence. To this end, wepropose an Attention-based Long Short-TermMemory Network for Aspect-level sentimentclassification. The attention mechanism canconcentrate on different parts of a sentencewhen different aspects are taken as input. Weexperiment on the SemEval 2014 dataset andresults show that our model achieves state-of-the-art performance on Aspect-level IntroductionSentiment analysis (Nasukawa and Yi, 2003), alsoknown as opinion mining (Liu, 2012), is a keyNLP task that receives much attention these Sentiment analysis is a fine-grainedtask that can provide complete and in-depth this paper, we deal with Aspect-level sentimentclassification and we find that the Sentiment polar-ity of a sentence is highly dependent on both con-tent and aspect.

3 For example, the Sentiment polarityof Staffs are not that friendly, but the taste coversall. will be positive if the aspect isfoodbut neg-ative when considering the aspectservice. Polaritycould be opposite when different aspects are networks have achieved state-of-the-artperformance in a variety of NLP tasks such as ma-chine translation (Lample et al., 2016), paraphraseidentification (Yin et al., 2015), question answer-ing (Golub and He, 2016) and text summariza-tion (Rush et al., 2015). However, neural net-work models are still in infancy to deal with Aspect-level Sentiment Classification . In some works, tar-get dependent Sentiment Classification can be ben-efited from taking into account target information,such as in Target-Dependent LSTM (TD-LSTM)and Target-Connection LSTM (TC-LSTM) (Tang etal.)

4 , 2015a). However, those models can only takeinto consideration the target but not aspect informa-tion which is proved to be crucial for has become an effective mechanism toobtain superior results, as demonstrated in imagerecognition (Mnih et al., 2014), machine transla-tion (Bahdanau et al., 2014), reasoning about entail-ment (Rockt aschel et al., 2015) and sentence sum-marization (Rush et al., 2015). Even more, neuralattention can improve the ability to read comprehen-sion (Hermann et al., 2015). In this paper, we pro-pose an attention mechanism to enforce the modelto attend to the important part of a sentence, in re-sponse to a specific aspect. We design an aspect-to-sentence attention mechanism that can concentrate606on the key part of a sentence given the explore the potential correlation of aspect andsentiment polarity in Aspect-level Sentiment classifi-cation.

5 In order to capture important information inresponse to a given aspect, we design an Attention-based LSTM. We evaluate our approach on a bench-mark dataset (Pontiki et al., 2014), which containsrestaurants and laptops main contributions of our work can be sum-marized as follows: We propose Attention-based Long Short-Termmemory for Aspect-level Sentiment classifica-tion. The models are able to attend differ-ent parts of a sentence when different aspectsare concerned. Results show that the attentionmechanism is effective. Since aspect plays a key role in this task, wepropose two ways to take into account aspectinformation during attention: one way is toconcatenate the aspect vector into the sentencehidden representations for computing attentionweights, and another way is to additionally ap-pend the aspect vector into the input word vec-tors.

6 Experimental results indicate that our ap-proach can improve the performance comparedwith several baselines, and further examplesdemonstrate the attention mechanism workswell for Aspect-level Sentiment rest of our paper is structured as follows:Section 2 discusses related works, Section 3 gives adetailed description of our Attention-based propos-als, Section 4 presents extensive experiments to jus-tify the effectiveness of our proposals, and Section 5summarizes this work and the future Related WorkIn this section, we will review related works onaspect-level Sentiment Classification and neural net-works for Sentiment Classification Sentiment Classification at Aspect-levelAspect-level Sentiment Classification is typicallyconsidered as a Classification problem in the liter-ature.

7 As we mentioned before, Aspect-level sen-timent Classification is a fine-grained classificationtask. The majority of current approaches attempt todetecting the polarity of the entire sentence, regard-less of the entities mentioned or aspects. Traditionalapproaches to solve those problems are to manu-ally design a set of features. With the abundance ofsentiment lexicons (Rao and Ravichandran, 2009;Perez-Rosas et al., 2012; Kaji and Kitsuregawa,2007), the lexicon -based features were built for sen-timent analysis (Mohammad et al., 2013). Most ofthese studies focus on building Sentiment classifierswith features, which include bag-of-words and sen-timent lexicons, using SVM (Mullen and Collier,2004). However, the results highly depend on thequality of features. In addition, feature engineeringis labor Sentiment Classification with NeuralNetworksSince a simple and effective approach to learn dis-tributed representations was proposed (Mikolov etal.)

8 , 2013), neural networks advance Sentiment anal-ysis substantially. Classical models including Re-cursive Neural Network (Socher et al., 2011; Donget al., 2014; Qian et al., 2015), Recursive Neu-ral Tensor Network (Socher et al., 2013), Recur-rent Neural Network (Mikolov et al., 2010; Tanget al., 2015b), LSTM (Hochreiter and Schmidhuber,1997) and Tree-LSTMs (Tai et al., 2015) were ap-plied into Sentiment analysis currently. By utilizingsyntax structures of sentences, tree-based LSTM shave been proved to be quite effective for many NLPtasks. However, such methods may suffer from syn-tax parsing errors which are common in resource-lacking has achieved a great success in variousNLP tasks. TD-LSTM and TC-LSTM (Tang etal., 2015a), which took target information into con-sideration, achieved state-of-the-art performancein target-dependent Sentiment Classification .

9 TC-LSTM obtained a target vector by averaging thevectors of words that the target phrase , simply averaging the word embeddings ofa target phrase is not sufficient to represent the se-mantics of the target phrase, resulting a the effectiveness of those methods, it isstill challenging to discriminate different sentimentpolarities at a fine-grained aspect level. Therefore,we are motivated to design a powerful neural net-work which can fully employ aspect information forsentiment Attention-based LSTM with Long Short-term Memory (LSTM)Recurrent Neural Network(RNN) is an extension ofconventional feed-forward neural network. How-ever, standard RNN has the gradient vanishingor exploding problems. In order to overcomethe issues, Long Short-term Memory network(LSTM) was developed and achieved superior per-formance (Hochreiter and Schmidhuber, 1997).

10 Inthe LSTM architecture, there are three gates and acell memory state. Figure 1 illustrates the architec-ture of a standard 1 2 1 2 Figure 1:The architecture of a standard LSTM.{w1,w2,..,wN}represent the word vector in a sen-tence whose length isN.{h1,h2,..,hN}is the formally, each cell in LSTM can be com-puted as follows:X=[ht 1xt](1)ft= (Wf X+bf)(2)it= (Wi X+bi)(3)ot= (Wo X+bo)(4)ct=ft ct 1+it tanh(Wc X+bc)(5)ht=ot tanh(ct)(6)whereWi; Wf; Wo2Rd 2dare the weighted ma-trices andbi; bf; bo2 Rdare biases of LSTM to belearned during training, parameterizing the transfor-mations of the input, forget and output gates respec-tively. is the sigmoid function and stands forelement-wise the inputsof LSTM cell unit, representing the word embed-ding vectorswtin Figure 1.