Distant supervision for relation extraction without ...

1 Distant supervision for relation extraction without labeled dataMike Mintz, Steven Bills, Rion Snow, Dan JurafskyStanford University / Stanford, CA models of relation extraction for tasks likeACE are based on supervised learning of relationsfrom small hand-labeled corpora. We investigate analternative paradigm that does not require labeledcorpora, avoiding the domain dependence of ACE-style algorithms, and allowing the use of corporaof any size. Our experiments use Freebase, a largesemantic database of several thousand relations, toprovidedistant supervision . For each pair of enti-ties that appears in some Freebase relation , we findall sentences containing those entities in a large un-labeled corpus and extract textual features to traina relation classifier. Our algorithm combines theadvantages of supervised IE (combining 400,000noisy pattern features in a probabilistic classifier)and unsupervised IE (extracting large numbers ofrelations from large corpora of any domain).

2 Ourmodel is able to extract 10,000 instances of 102 re-lations at a precision of We also analyzefeature performance, showing that syntactic parsefeatures are particularly helpful for relations that areambiguous or lexically Distant in their IntroductionAt least three learning paradigms have been ap-plied to the task of extracting relational facts fromtext (for example, learning that a person is em-ployed by a particular organization, or that a ge-ographic entity is located in a particular region).In supervised approaches, sentences in a cor-pus are first hand-labeled for the presence of en-tities and the relations between them. The NISTA utomatic Content extraction (ACE) RDC 2003and 2004 corpora, for example, include over 1,000documents in which pairs of entities have been la-beled with 5 to 7 major relation types and 23 to24 subrelations, totaling 16,771 relation systems then extract a wide variety of lexi-cal, syntactic, and semantic features, and use su-pervised classifiers to label therelation mentionholding between a given pair of entities in a testset sentence, optionally combining relation men-tions (Zhou et al.)

3 , 2005; Zhou et al., 2007; Sur-deanu and Ciaramita, 2007).Supervised relation extraction suffers from anumber of problems, however. Labeled trainingdata is expensive to produce and thus limited inquantity. Also, because the relations are labeledon a particular corpus, the resulting classifiers tendto be biased toward that text alternative approach, purely unsupervisedinformation extraction , extracts strings of wordsbetween entities in large amounts of text, andclusters and simplifies these word strings to pro-duce relation -strings (Shinyama and Sekine, 2006;Banko et al., 2007). Unsupervised approaches canuse very large amounts of data and extract verylarge numbers of relations, but the resulting rela-tions may not be easy to map to relations neededfor a particular knowledge third approach has been to use a very smallnumber of seed instances or patterns to do boot-strap learning (Brin, 1998; Riloff and Jones, 1999;Agichtein and Gravano, 2000; Ravichandran andHovy, 2002; Etzioni et al.

4 , 2005; Pennacchiottiand Pantel, 2006; Bunescu and Mooney, 2007;Rozenfeld and Feldman, 2008). These seeds areused with a large corpus to extract a new set ofpatterns, which are used to extract more instances,which are used to extract more patterns, in an it-erative fashion. The resulting patterns often sufferfrom low precision and semantic propose an alternative paradigm, Distant su-pervision, that combines some of the advantagesof each of these approaches. Distant supervisionis an extension of the paradigm used by Snow etal. (2005) for exploiting WordNet to extracthyper-nym(is-a) relations between entities, and is simi-lar to the use ofweakly labeled datain bioinfor-matics (Craven and Kumlien, 1999; Morgan et al., relation nameNew instance/location/location/containsParis , Montmartre/location/location/containsOnt ario, Fort Erie/music/artist/originMighty Wagon, Cincinnati/people/deceasedperson/placeof deathFyodor Kamensky, Clearwater/people/person/nationalityMari anne Yvonne Heemskerk, Netherlands/people/person/placeofbirthWa vell Wayne Hinds, Kingston/book/author/workswrittenUpton Sinclair, Lanny Budd/business/company/foundersWWE, Vince McMahon/people/person/professionThomas Mellon, judgeTable 1: Ten relation instances extracted by our system that did not appear in ).

5 Our algorithm uses Freebase (Bollacker etal., 2008), a large semantic database, to providedistant supervision for relation extraction . Free-base contains 116 million instances of 7,300 rela-tions between 9 million entities. The intuition ofdistant supervision is that any sentence that con-tains a pair of entities that participate in a knownFreebase relation is likely to express that relationin some way. Since there may be many sentencescontaining a given entity pair, we can extract verylarge numbers of (potentially noisy) features thatare combined in a logistic regression whereas the supervised training paradigmuses a small labeled corpus of only 17,000 rela-tion instances as training data, our algorithm canuse much larger amounts of data: more text, morerelations, and more instances. We use millionWikipedia articles and million instances of 102relations connecting 940,000 entities. In addition,combining vast numbers of features in a large clas-sifier helps obviate problems with bad our algorithm is supervised by adatabase, rather than by labeled text, it doesnot suffer from the problems of overfitting anddomain-dependence that plague supervised sys-tems.

6 supervision by a database also means that,unlike in unsupervised approaches, the output ofour classifier uses canonical names for paradigm offers a natural way of integratingdata from multiple sentences to decide if a relationholds between two entities. Because our algorithmcan use large amounts of unlabeled data, a pair ofentities may occur multiple times in the test each pair of entities, we aggregate the featuresfrom the many different sentences in which thatpair appeared into a single feature vector, allowingus to provide our classifier with more information,resulting in more accurate 1 shows examples of relation instancesextracted by our system. We also use this systemto investigate the value of syntactic versus lexi-cal (word sequence) features in relation syntactic features are known to improve theperformance of supervised IE, at least using cleanhand-labeled ACE data (Zhou et al., 2007; Zhouet al.)

7 , 2005), we do not know whether syntacticfeatures can improve the performance of unsuper-vised or distantly supervised IE. Most previousresearch in bootstrapping or unsupervised IE hasused only simple lexical features, thereby avoid-ing the computational expense of parsing (Brin,1998; Agichtein and Gravano, 2000; Etzioni et al.,2005), and the few systems that have used unsu-pervised IE have not compared the performanceof these two types of Previous workExcept for the unsupervised algorithms discussedabove, previous supervised or bootstrapping ap-proaches to relation extraction have typically re-lied on relatively small datasets, or on only a smallnumber of distinct relations. Approaches based onWordNet have often only looked at the hypernym(is-a) or meronym (part-of) relation (Girju et al.,2003; Snow et al., 2005), while those based on theACE program (Doddington et al., 2004) have beenrestricted in their evaluation to a small number ofrelation instances and corpora of less than a mil-lion early algorithms for relation extractionused little or no syntactic information.

8 For ex-ample, the DIPRE algorithm by Brin (1998) usedstring-based regular expressions in order to rec-ognize relations such asauthor-book, while theSNOWBALL algorithm by Agichtein and Gravano(2000) learned similar regular expression patternsover words and named entity tags. Hearst (1992)used a small number of regular expressions overwords and part-of-speech tags to find examples ofthe hypernym relation . The use of these patternshas been widely replicated in successful systems,for example by Etzioni et al. (2005). Other workRelation nameSizeExample/people/person/nationalit y281,107 John Dugard, South Africa/location/location/contains253,223 Belgium, Nijlen/people/person/profession208,888 Dusa McDuff, Mathematician/people/person/placeofbirth 105,799 Edwin Hubble, Marshfield/dining/restaurant/cuisine86,2 13 MacAyo s Mexican Kitchen, Mexican/business/businesschain/location6 6,529 Apple Inc., Apple Inc., South Park, NC/biology/organismclassificationrank42, 806 Scorpaeniformes, Order/film/film/genre40,658 Where the Sidewalk Ends, Film noir/film/film/language31,103 Enter the Phoenix, Cantonese/biology/organismhigherclassifi cation30,052 Calopteryx, Calopterygidae/film/film/country27,217 Turtle Diary, United States/film/writer/film23,856 Irving Shulman, Rebel without a Cause/film/director/film23,539 Michael Mann, Collateral/film/producer/film22,079 Diane Eskenazi, Aladdin/people/deceasedperson/placeofdea th18,814 John W.

9 Kern, Asheville/music/artist/origin18,619 The Octopus Project, Austin/people/person/religion17,582 Joseph Chartrand, Catholicism/book/author/workswritten17,2 78 Paul Auster, Travels in the Scriptorium/soccer/footballposition/play ers17,244 Midfielder, Chen Tao/people/deceasedperson/causeofdeath16 ,709 Richard Daintree, Tuberculosis/book/book/genre16,431 Pony Soldiers, Science fiction/film/film/music14,070 Stavisky, Stephen Sondheim/business/company/industry13,805 ATS Medical, Health careTable 2: The 23 largest Freebase relations we use, with their size and an instance of each as Ravichandran and Hovy (2002) and Pan-tel and Pennacchiotti (2006) use the same formal-ism of learning regular expressions over words andpart-of-speech tags to discover patterns indicatinga variety of recent approaches have used deeper syn-tactic information derived from parses of the inputsentences, including work exploiting syntactic de-pendencies by Lin and Pantel (2001) and Snow etal.

10 (2005), and work in the ACE paradigm suchas Zhou et al. (2005) and Zhou et al. (2007).Perhaps most similar to our Distant supervisionalgorithm is the effective method of Wu and Weld(2007) who extract relations from a Wikipediapage by using supervision from the page their corpus-specific method, which is spe-cific to a (single) Wikipedia page, our algorithmallows us to extract evidence for a relation frommany different documents, and from any FreebaseFollowing the literature, we use the term rela-tion to refer to an ordered, binary relation be-tween entities. We refer to individual ordered pairsin this relation as relation instances . For ex-ample, theperson-nationalityrelation holds be-tween the entities named John Steinbeck and United States , so it has John Steinbeck,United States as an use relations and relation instances fromFreebase, a freely available online database ofstructured semantic in Freebase iscollected from a variety of sources.