A Joint Neural Model for Information Extraction with ...

1 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999 8009 July 5 - 10, 2020 Association for Computational Linguistics7999A Joint Neural Model for Information Extraction with Global FeaturesYing Lin1, Heng Ji1, Fei Huang2, Lingfei Wu31 University of Illinois at Urbana-Champaign2 Alibaba DAMO Academy3 IBM existing Joint Neural models for Infor-mation Extraction (IE) use local task-specificclassifiers to predict labels for individual in-stances ( , trigger, relation) regardless oftheir interactions. For example, aVICTIMofaDIEevent is likely to be aVICTIMof anAT-TACK event in the same sentence. In order tocapture such cross-subtask and cross-instanceinter-dependencies, we propose a Joint neuralframework, ONEIE, that aims to extract theglobally optimal IE result as a graph from aninput sentence.

2 ONEIE performs end-to-endIE in four stages: (1) Encoding a given sen-tence as contextualized word representations;(2) Identifying entity mentions and event trig-gers as nodes; (3) Computing label scores forall nodes and their pairwise links using localclassifiers; (4) Searching for the globally op-timal graph with a beam decoder. At the de-coding stage, we incorporate global featuresto capture the cross-subtask and cross-instanceinteractions. Experiments show that addingglobal features improves the performance ofour Model and achieves new state-of-the-arton all subtasks. As ONEIE does not use anylanguage-specific feature, we prove it can beeasily applied to new languages or trained ina multilingual manner. Our code and modelsfor English, Spanish and Chinese are publiclyavailable for research IntroductionInformation Extraction (IE) aims to extract struc-tured Information from unstructured texts .

3 It is acomplex task comprised of a wide range of sub-tasks, such as named, nominal, and pronominalmention Extraction , entity linking, entity corefer-ence resolution, relation Extraction , event extrac-tion, and event coreference resolution. Early ef-forts typically perform IE in a pipelined fashion,1 leads to the error propagation problem anddisallows interactions among components in thepipeline. As a solution, some researchers proposejoint inference and Joint modeling methods to im-prove local prediction (Roth and Yih, 2004; Ji andGrishman, 2005; Ji et al., 2005; Sil and Yates, 2013;Li et al., 2014; Durrett and Klein, 2014; Miwaand Sasaki, 2014; Lu and Roth, 2015; Yang andMitchell, 2016; Kirschnick et al., 2016). Due tothe success of deep learning, Neural models havebeen widely applied to various IE subtasks (Col-lobert et al.)

4 , 2011; Chiu and Nichols, 2016; Chenet al., 2015; Lin et al., 2016). Recently, some ef-forts (Wadden et al., 2019; Luan et al., 2019) re-visit global inference approaches by designing neu-ral networks with embedding features to jointlymodel multiple subtasks. However, these methodsuse separate local task-specific classifiers in thefinal layer and do not explicitly Model the inter-dependencies among tasks and instances. Figure 1shows a real example where the local argument roleclassifier predicts a redundantPERSON edge. Themodel should be able to avoid such mistakes if it iscapable of learning and leveraging the fact that it isunusual for anELECT event to have GulEnd-PositionresignedElectwonpersonper sonExample: Prime Minister Abdullah Gul resigned earlierTuesday to make way for Erdogan, who won aparliamentary seat in by-elections 1: A typical error made by local classifiers with-out global address this issue, we propose a Joint neu-8000earthquakekilledThe19peopleandin jured300inKashmirregion,IndiaIdentificat ionClassificationTriggerEntityRoleRelati onDecodingEncodingDiePERvictimInjureDieP ERvictimInjureDiePERvictimInjureORGPER victimvictim Injure-victim-ORG searchScore vectorsInformation networkFigure 2.

5 An illustration of our end-to-end Joint Information Extraction framework ONEIE at the test stage. We donot show all pairwise links for simplicity framework,ONEIE, to perform end-to-end IEwith global constraints. As Figure 2 shows, in-stead of predicting separate knowledge elementsusing local classifiers,ONEIE aims to extract aglobally optimal Information network for the in-put sentence. When comparing candidate infor-mation networks during the decoding process, wenot only consider individual label scores for eachknowledge element, but evaluate cross-subtask andcross-instance interactions in the network. In thisexample, a graph with theINJURE-VICTIM-ORG(theVICTIMof anINJURE event is anORGentity)structure is demoted. Experiments show that ourframework achieves comparable or better resultscompared to the state-of-the-art end-to-end archi-tecture (Wadden et al.)

6 , 2019).To the best of our knowledge,ONEIEis thefirst end-to-end Neural IE framework that explic-itly models cross-subtask and cross-instance inter-dependencies and predicts the result as a unifiedgraph instead of isolated knowledge elements. Be-causeONEIE does not rely on language-specificfeatures, it can be rapidly applied to new , global features in our framework arehighly explainable and can be explicitly TaskGiven a sentence, ourONEIE framework aims toextract aninformation networkrepresentation (Liet al., 2014), where entity mentions and event trig-gers are represented as nodes, and relations andevent-argument links are represented as edges. Inother words, we perform entity, relation, and eventextraction within a unified framework. In this sec-tion, we will elaborate these tasks and Extractionaims to identify entity men-tions in text and classify them into pre-defined en-tity types.

7 A mention can be a name, nominal, orpronoun. For example, Kashmir region shouldbe recognized as a location(LOC) named entitymention in Figure Extractionis the task of assigning arelation type to an ordered pair of entity example, there is aPART-WHOLE relation be-tween Kashmir region and India .Event Extractionentails identifyingevent trig-gers(the words or phrases that most clearly ex-press event occurrences) and theirarguments(thewords or phrases for participants in those events)in unstructured texts and classifying these phrases,respectively, for their types and roles. An argumentcan be an entity, time expression, or value ( ,MONEY,JOB-TITLE,CRIME). For example, in Fig-ure 2, the word injured triggers anINJURE eventand 300 is formulate the task of extracting informationnetworks as follows.

8 Given an input sentence, ourgoal is to predict a graphG= (V,E), whereVandEare the node and edge sets respectively. Eachnodevi= ai,bi,li Vrepresents an entity men-tion or event trigger, whereaandbare the startand end word indices, andlis the node type la-bel. Each edgeeij= i,j,lij Eis representedsimilarly, whereasiandjdenote the indices of in-volved nodes. For example, in Figure 2, the trigger injured is represented as 7, 7,INJURE , the entitymention Kashmir region is represented as 10,800111,LOC , and the event-argument edge betweenthem is 2, 3,PLACE .3 ApproachAs Figure 2 illustrates, ourONEIE framework ex-tracts the Information network from a given sen-tence in four steps: encoding, identification, clas-sification, and decoding. We encode the input sen-tence using a pre-trained BERT encoder (Devlinet al.)

9 , 2019) and identify entity mentions and eventtriggers in the sentence. After that, we computethe type label scores for all nodes and pairwiseedges among them. During decoding, we explorepossible Information networks for the input sen-tence using beam search and return the one withthe highest global EncodingGiven an input sentence ofLwords, we obtainthe contextualized representationxifor each wordusing a pre-trained BERT encoder. If a word issplit into multiple word pieces ( , Mondrian Mon,##dr,##ian), we use the average of all piecevectors as its word representation. While previousmethods typically use the output of the last layer ofBERT, our preliminary study shows that enrichingword representations using the output of the thirdlast layer of BERT can substantially improve theperformance on most IdentificationAt this stage, we identify entity mentions andevent triggers in the sentence, which will act asnodes in the Information network.

10 We use a feed-forward networkFFNto compute a score vector yi= FFN(xi)for each word, where each value in yirepresents the score for a tag in a target tagset2. After that, we use a conditional randomfields (CRFs) layer to capture the dependenciesbetween predicted tags ( , anI-PERtag shouldnot follow aB-GPEtag). Similar to (Chiu andNichols, 2016), we calculate the score of a tag path z={ z1,.., zL}ass(X, z) =L i=1 yi, zi+L+1 i=1A zi 1, zi,whereX={x1,..,xL}is the contextualized rep-resentations of the input sequence, yi, ziis the zi-th2We use the BIO tag scheme, in which the prefixB- marksthe beginning of a mention, andI- means inside of a token not belonging to any mention is tagged of the score vector yi, andA zi 1, ziisthe( zi 1, zi)entry in matrixAthat indicates thetransition score from tag zi 1to zi.