Neural Architectures for Named Entity Recognition

1 Proceedings of NAACL-HLT 2016, pages 260 270,San Diego, California, June 12-17, 2016 Association for Computational LinguisticsNeural Architectures for Named Entity RecognitionGuillaume Lample Miguel Ballesteros Sandeep Subramanian Kazuya Kawakami Chris Dyer Carnegie Mellon University NLP Group, Pompeu Fabra Named Entity Recognition sys-tems rely heavily on hand-crafted features anddomain-specific knowledge in order to learneffectively from the small, supervised trainingcorpora that are available. In this paper, weintroduce two new Neural Architectures onebased on bidirectional LSTMs and conditionalrandom fields, and the other that constructsand labels segments using a transition-basedapproach inspired by shift-reduce models rely on two sources of infor-mation about words: character-based wordrepresentations learned from the supervisedcorpus and unsupervised word representa-tions learned from unannotated corpora.

2 Ourmodels obtain state-of-the-art performance inNER in four languages without resorting toany language-specific knowledge or resourcessuch as IntroductionNamed Entity Recognition (NER) is a challenginglearning problem. One the one hand, in most lan-guages and domains, there is only a very smallamount of supervised training data available. On theother, there are few constraints on the kinds of wordsthat can be names, so generalizing from this smallsample of data is difficult. As a result, carefully con-structed orthographic features and language-specificknowledge resources, such as gazetteers, are widelyused for solving this task.

3 Unfortunately, language-specific resources and features are costly to de-velop in new languages and new domains, makingNER a challenge to adapt. Unsupervised learning1 The code of the LSTM-CRF and Stack-LSTM unannotated corpora offers an alternative strat-egy for obtaining better generalization from smallamounts of supervision. However, even systemsthat have relied extensively on unsupervised fea-tures (Collobert et al., 2011; Turian et al., 2010;Lin and Wu, 2009; Ando and Zhang, 2005b,in-ter alia) have used these to augment, rather thanreplace, hand-engineered features ( , knowledgeabout capitalization patterns and character classes ina particular language) and specialized knowledge re-sources ( , gazetteers).

4 In this paper, we present Neural architecturesfor NER that use no language-specific resourcesor features beyond a small amount of supervisedtraining data and unlabeled mod-els are designed to capture two intuitions. First,since names often consist of multiple tokens, rea-soning jointly over tagging decisions for each to-ken is important. We compare two models here,(i) a bidirectional LSTM with a sequential condi-tional random layer above it (LSTM-CRF; 2), and(ii) a new model that constructs and labels chunksof input sentences using an algorithm inspired bytransition-based parsing with states represented bystack LSTMs (S-LSTM; 3).

5 second , token-levelevidence for being a name includes both ortho-graphic evidence (what does the word being taggedas a name look like?) and distributional evidence(where does the word being tagged tend to oc-cur in a corpus?). To capture orthographic sen-sitivity, we use character-based word representa-tion model (Ling et al., 2015b) to capture distribu-tional sensitivity, we combine these representationswith distributional representations (Mikolov et al.,2013b).

6 Our word representations combine both ofthese, and dropout training is used to encourage themodel to learn to trust both sources of evidence ( 4).Experiments in English, Dutch, German, andSpanish show that we are able to obtain state-260of-the-art NER performance with the LSTM-CRFmodel in Dutch, German, and Spanish, and verynear the state-of-the-art in English without anyhand-engineered features or gazetteers ( 5). Thetransition-based algorithm likewise surpasses thebest previously published results in several lan-guages, although it performs less well than theLSTM-CRF LSTM-CRF ModelWe provide a brief description of LSTMs and CRFs,and present a hybrid tagging architecture .

7 This ar-chitecture is similar to the ones presented by Col-lobert et al. (2011) and Huang et al. (2015). LSTMR ecurrent Neural networks (RNNs) are a familyof Neural networks that operate on take as input a sequence of vectors(x1,x2,..,xn)and return another sequence(h1,h2,..,hn)that represents some informationabout the sequence at every step in the RNNs can, in theory, learn long depen-dencies, in practice they fail to do so and tend tobe biased towards their most recent inputs in thesequence (Bengio et al.)

8 , 1994). Long Short-termMemory Networks (LSTMs) have been designed tocombat this issue by incorporating a memory-celland have been shown to capture long-range depen-dencies. They do so using several gates that controlthe proportion of the input to give to the memorycell, and the proportion from the previous state toforget (Hochreiter and Schmidhuber, 1997). We usethe following implementation:it= (Wxixt+Whiht 1+Wcict 1+bi)ct= (1 it) ct 1+it tanh(Wxcxt+Whcht 1+bc)ot= (Wxoxt+Whoht 1+Wcoct+bo)ht=ot tanh(ct),where is the element-wise sigmoid function, and is the element-wise a given sentence(x1,x2.

9 ,xn)containingnwords, each represented as ad-dimensional vector,an LSTM computes a representation htof the leftcontext of the sentence at every wordt. Naturally,generating a representation of the right context htas well should add useful information. This can beachieved using a second LSTM that reads the samesequence in reverse. We will refer to the former asthe forward LSTM and the latter as the backwardLSTM. These are two distinct networks with differ-ent parameters. This forward and backward LSTM pair is referred to as a bidirectional LSTM (Gravesand Schmidhuber, 2005).

10 The representation of a word using this model isobtained by concatenating its left and right contextrepresentations,ht= [ ht; ht]. These representa-tions effectively include a representation of a wordin context, which is useful for numerous tagging CRF Tagging ModelsA very simple but surprisingly effective taggingmodel is to use theht s as features to make indepen-dent tagging decisions for each outputyt(Ling etal., 2015b). Despite this model s success in simpleproblems like POS tagging, its independent classifi-cation decisions are limiting when there are strongdependencies across output labels.