Attention-Based Bidirectional Long Short-Term Memory ...

1 Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 207 212,Berlin, Germany, August 7-12, 2016 Association for Computational LinguisticsAttention-Based Bidirectional Long Short-Term Memory Networks forRelation ClassificationPeng Zhou, Wei Shi, Jun Tian, Zhenyu Qi , Bingchen Li, Hongwei Hao, Bo XuInstitute of Automation, Chinese Academy of Sciences{zhoupeng2013, shiwei2013, tianjun2013, ,libingchen2013, , classification is an important se-mantic processing task in the field of nat-ural language processing (NLP). State-of-the-art systems still rely on lexical re-sources such as WordNet or NLP systemslike dependency parser and named entityrecognizers (NER) to get high-level fea-tures. Another challenge is that importantinformation can appear at any position inthe sentence. To tackle these problems,we propose Attention-Based BidirectionalLong Short-Term Memory Networks(Att-BLSTM) to capture the most important se-mantic information in a sentence.}

2 The ex-perimental results on the SemEval-2010relation classification task show that ourmethod outperforms most of the existingmethods, with only word IntroductionRelation classification is the task of finding seman-tic relations between pairs of nominals, which isuseful for many NLP applications, such as infor-mation extraction (Wu and Weld, 2010), questionanswering (Yao and Van Durme, 2014). For in-stance, the following sentence contains an exam-ple of the Entity-Destination relation between thenominalsFlowersandchapel. e1 Flowers /e1 are carried into the e2 chapel /e2 . e1 , /e1 , e2 , /e2 are four position indica-tors which specify the starting and ending of thenominals (Hendrickx et al., 2009).Traditional relation classification methods thatemploy handcrafted features from lexical re-sources, are usually based on pattern matching,and have achieved high performance (Bunescu Correspondence author: Mooney, 2005; Mintz et al.)

3 , 2009; Rink andHarabagiu, 2010). One downside of these meth-ods is that many traditional NLP systems are uti-lized to extract high-level features, such as part ofspeech tags, shortest dependency path and namedentities, which consequently results in the increaseof computational cost and additional propagationerrors. Another downside is that designing fea-tures manually is time-consuming, and performingpoor on generalization due to the low coverage ofdifferent training , deep learning methods provide an ef-fective way of reducing the number of handcraftedfeatures (Socher et al., 2012; Zeng et al., 2014).However, these approaches still use lexical re-sources such as WordNet (Miller, 1995) or NLPsystems like dependency parsers and NER to gethigh-level paper proposes a novel neural network Att-BLSTM for relation classification. Our model uti-lizes neural attention mechanism with Bidirection-al Long Short-Term Memory Networks(BLSTM)to capture the most important semantic informa-tion in a sentence.

4 This model doesn t utilize anyfeatures derived from lexical resources or contribution of this paper is using BLST-M with attention mechanism, which can automat-ically focus on the words that have decisive effecton classification, to capture the most important se-mantic information in a sentence, without usingextra knowledge and NLP systems. We conductexperiments on the SemEval-2010 Task 8 dataset,and achieve anF1-score , higher thanmost of the existing methods in the remainder of the paper is structured as fol-lows. In Section 2, we review related work aboutrelation classification. Section 3 presents our Att-BLSTM model in detail. In Section 4, we describedetails about the setup of experimental evaluation207and the experimental results. Finally, we have ourconclusion in Section Related WorkOver the years, various methods have been pro-posed for relation classification. Most of them arebased on pattern matching and apply extra NLPsystems to derive lexical features.

5 One relatedwork is proposed by Rink and Harabagiu (2010),which utilizes many features derived from exter-nal corpora for a Support Vector Machine(SVM) , deep neural networks can learn under-lying features automatically and have been usedin the literature. Most representative progresswas made by Zeng et al. (2014), who utilizedconvolutional neural networks(CNN) for relationclassification. While CNN is not suitable forlearning long-distance semantic information, soour approach builds on Recurrent neural Net-work(RNN) (Mikolov et al., 2010).One related work was proposed by Zhang andWang (2015), which employed Bidirectional RN-N to learn patterns of relations from raw text da-ta. Although Bidirectional RNN has access toboth past and future context information, the rangeof context is limited due to the vanishing gradi-ent problem. To overcome this problem, Longshort-Term Memory (LSTM) units are introducedby Hochreiter and Schmidhuber (1997).

6 Another related work is SDP-LSTM model pro-posed by Yan et al. (2015). This model leveragesthe shortest dependency path(SDP) between twonominals, then it picks up heterogeneous informa-tion along the SDP with LSTM units. While ourmethod regards the raw text as a , our work is related to BLSTM mod-el proposed by Zhang et al. (2015). This mod-el utilizing NLP tools and lexical resources toget word, position, POS, NER, dependency parseand hypernym features, together with LSTM u-nits, achieved a comparable result to the state-of-the-art. However, comparing to the complicatedfeatures that employed by Zhang et al. (2015),our method regards the four position indicators e1 , /e1 , e2 , /e2 as single words, and trans-forms all words to word vectors, forming a simplebut competing ModelIn this section we propose Att-BLSTM model indetail. As shown in Figure 1, the model proposedin this paper contains five components:(1) Input layer: input sentence to this model;(2) Embedding layer: map each word into a lowdimension vector;(3) LSTM layer: utilize BLSTM to get high lev-el features from step (2);(4) Attention layer: produce a weight vector,and merge word-level features from each time stepinto a sentence-level feature vector, by multiplyingthe weight vector;(5) Output layer: the sentence-level feature vec-tor is finally used for relation components will be presented in detail inthis Word EmbeddingsGiven a sentence consisting ofTwordsS={x1, x2.}

7 , xT}, every wordxiis converted in-to a real-valued vectorei. For each word inS,we first look up the embedding matrixWwrd Rdw|V|, whereVis a fixed-sized vocabulary, anddwis the size of word embedding. The matrixWwrdis a parameter to be learned, anddwis ahyper-parameter to be chosen by user. We trans-form a wordxiinto its word embeddingeiby us-ing the matrix-vector product:ei=Wwrdvi(1)whereviis a vector of size|V|which has value1 at indexeiand 0 in all other positions. Then thesentence is feed into the next layer as a real-valuedvectorsembs={e1, e2, .. , eT}. Bidirectional NetworkLSTM units are firstly proposed by Hochreiter andSchmidhuber (1997) to overcome gradient vanish-ing problem. The main idea is to introduce anadaptive gating mechanism, which decides the de-gree to which LSTM units keep the previous s-tate and memorize the extracted features of thecurrent data input. Then lots of LSTM variantshave been proposed.

8 We adopt a variant intro-duced by Graves et al. (2013), which adds weight-ed peephole connections from theConstant ErrorCarousel(CEC) to the gates of the same memoryblock. By directly employing the current cell stateto generate the gate degrees, the peephole connec-tions allow all gates toinspectinto the cell ( +1x1e1h 1h1h TxTeTh ThTh 2x2e2h 2h2h 3x3e3h 3h3h \InputLayerEmbeddingLayerLSTML ayerAttentionLayerOutputLayer..Figure 1: Bidirectional LSTM model with Attentionthe current cell state) even when the output gate isclosed (Graves, 2013).Typically, four components composite theLSTM-based recurrent neural networks: one in-put gateitwith corresponding weight matrixWxi, Whi, Wci, bi; one forget gateftwith corre-sponding weight matrixWxf, Whf, Wcf, bf; oneoutput gateotwith corresponding weight matrixWxo, Who, Wco, bo, all of those gates are set togenerate some degrees, using current inputxi, thestatehi 1that previous step generated , and cur-rent state of this cellci 1(peephole), for the deci-sions whether to take the inputs, forget the memo-ry stored before, and output the state generated lat-er.

9 Just as these following equations demonstrate:it= (Wxixt+Whiht 1+Wcict 1+bi)(2)ft= (Wxfxt+Whfht 1+Wcfct 1+bf)(3)gt= tanh(Wxcxt+Whcht 1+Wccct 1+bc)(4)ct=itgt+ftct 1(5)ot= (Wxoxt+Whoht 1+Wcoct+bo)(6)ht=ottanh(ct)(7)Hence, current cell statectwill be generated bycalculating the weighted sum using both previouscell state and current information generated by thecell (Graves, 2013).For many sequence modelling tasks, it is benefi-cial to have access to future as well as past contex-t. However, standard LSTM networks process se-quences in temporal order, they ignore future con-text. Bidirectional LSTM networks extend the uni-directional LSTM networks by introducing a sec-ond layer, where the hidden to hidden connection-s flow in opposite temporal order. The model istherefore able to exploit information both from thepast and the this paper, we use BLSTM. As also shown inFigure 1, the network contains two sub-networksfor the left and right sequence context, which areforward and backward pass respectively.

10 The out-put of theithword is shown in the following equa-tion:hi= [ hi hi](8)Here, we use element-wise sum to combine theforward and backward pass AttentionAttentive neural networks have recently demon-strated success in a wide range of tasks rangingfrom question answering, machine translations,speech recognition, to image captioning (Herman-n et al., 2015; Bahdanau et al., 2014; Chorows-ki et al., 2015; Xu et al., 2015). In this section,we propose the attention mechanism for relationclassification tasks. LetHbe a matrix consistingof output vectors[h1, h2, .. , hT]that the LSTM layer produced, whereTis the sentence representationrof the sentence is formed bya weighted sum of these output vectors:M= tanh(H)(9) =sof tmax(wTM)(10)r=H T(11)209 ModelFeature SetF1 SVMPOS, prefixes, morphological, WordNet, dependency parse,(Rink and Harabagiu, 2010)Levin classed, ProBank, FramNet, NomLex-Plus, n-gram, paraphrases, TextRunnerCNNWV (Turian et al.)