Learning Entity and Relation Embeddings for Knowledge ...

1 Learning Entity and Relation Embeddings for Knowledge Graph Completion Yankai Lin1 , Zhiyuan Liu1 , Maosong Sun1,2 , Yang Liu3 , Xuan Zhu3. 1. Department of Computer Science and Technology, State Key Lab on Intelligent Technology and Systems, National Lab for Information Science and Technology, Tsinghua University, Beijing, China 2. Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China 3. Samsung R&D Institute of China, Beijing, China Abstract For this reason, traditional approach of link prediction is not capable for Knowledge graph completion. Recently, Knowledge graph completion aims to perform link pre- diction between entities. In this paper, we consider the a promising approach for the task is embedding a knowl- approach of Knowledge graph Embeddings . Recently, edge graph into a continuous vector space while preserving models such as TransE and TransH build Entity and re- certain information of the graph.

2 Following this approach, lation Embeddings by regarding a Relation as translation many methods have been explored, which will be introduced from head Entity to tail Entity . We note that these model- in detail in Section Related Work . s simply put both entities and relations within the same Among these methods, TransE (Bordes et al. 2013) and semantic space. In fact, an Entity may have multiple as- TransH (Wang et al. 2014) are simple and effective, achiev- pects and various relations may focus on different as- ing the state-of-the-art prediction performance. TransE, in- pects of entities, which makes a common space insuf- spired by (Mikolov et al. 2013b), learns vector Embeddings ficient for modeling. In this paper, we propose Tran- sR to build Entity and Relation Embeddings in separate for both entities and relationships.

3 These vector Embeddings Entity space and Relation spaces. Afterwards, we learn are set in Rk and we denote with the same letters in bold- Embeddings by first projecting entities from Entity s- face. The basic idea behind TransE is that, the relationship pace to corresponding Relation space and then building between two entities corresponds to a translation between translations between projected entities. In experiments, the Embeddings of entities, that is, h + r t when (h, r, t). we evaluate our models on three tasks including link holds. Since TransE has issues when modeling 1-to-N, N- prediction, triple classification and relational fact ex- to-1 and N-to-N relations, TransH is proposed to enable an traction. Experimental results show significant and con- Entity having different representations when involved in var- sistent improvements compared to state-of-the-art base- ious relations.

4 Lines including TransE and TransH. The source code of this paper can be obtained from https://github. Both TransE and TransH assume Embeddings of entities com/mrlyk423/relation_extraction. and relations being in the same space Rk . However, an enti- ty may have multiple aspects, and various relations focus on different aspects of entities. Hence, it is intuitive that some Introduction entities are similar and thus close to each other in the enti- Knowledge graphs encode structured information of entities ty space, but are comparably different in some specific as- and their rich relations. Although a typical Knowledge graph pects and thus far away from each other in the correspond- may contain millions of entities and billions of Relation - ing Relation spaces. To address this issue, we propose a new al facts, it is usually far from complete.

5 Knowledge graph method, which models entities and relations in distinct s- completion aims at predicting relations between entities un- paces, , Entity space and multiple Relation spaces ( , der supervision of the existing Knowledge graph. Knowledge Relation -specific Entity spaces), and performs translation in graph completion can find new relational facts, which is an the corresponding Relation space, hence named as TransR. important supplement to Relation extraction from plain texts. The basic idea of TransR is illustrated in Fig. 1. For each Knowledge graph completion is similar to link prediction triple (h, r, t), entities in the Entity space are first projected in social network analysis, but more challenging for the fol- into r- Relation space as hr and tr with operation Mr , and lowing reasons: (1) nodes in Knowledge graphs are entities then hr + r tr.

6 The Relation -specific projection can make with different types and attributes; and (2) edges in knowl- the head/tail entities that actually hold the Relation (denoted edge graphs are relations of different types. For Knowledge as colored circles) close with each other, and also get far graph completion, we not only determine whether there is away from those that do not hold the Relation (denoted as a Relation between two entities or not, but also predict the colored triangles). specific type of the Relation . Moreover, under a specific Relation , head-tail Entity pairs . Corresponding author: Zhiyuan Liu usually exhibit diverse patterns. It is insufficient to build on- Copyright c 2015, Association for the Advancement of Artificial ly a single Relation vector to perform all translations from Intelligence ( ).

7 All rights reserved. head to tail entities. For example, the head-tail entities of also be compared as baselines with our models in experi- ments. Unstructured Model (UM). UM model (Bordes et al. 2012; 2014) was proposed as a naive version of TransE by assigning all r = 0, leading to score function fr (h, t) =. kh tk22 . This model cannot consider differences of rela- tions. Structured Embedding (SE). SE model (Bordes et al. Entity Space Relation Space of r 2011) designs two Relation -specific matrices for head and tail entities, , Mr,1 and Mr,2 , and defines the score func- Figure 1: Simple illustration of TransR. tion as an L1 distance between two projected vectors, , fr (h, t) = kMr,1 h Mr,2 tk1 . Since the model has two separate matrices for optimization, it cannot capture precise the Relation location location contains have many pattern- relations between entities and relations.

8 S such as country-city, country-university, continent-country Single Layer Model (SLM). SLM model was proposed and so on. Following the idea of piecewise linear regression as a naive baseline of NTN (Socher et al. 2013). The score (Ritzema and others 1994), we extend TransR by clustering function of SLM model is defined as diverse head-tail Entity pairs into groups and Learning dis- tinct Relation vectors for each group, named as cluster-based fr (h, t) = u>. r g(Mr,1 h + Mr,2 t), (3). TransR (CTransR). We evaluate our models with the tasks of link prediction, where Mr,1 and Mr,2 are weight matrices, and g() is the triple classification and Relation fact extraction on bench- tanh operation. SLM is a special case of NTN when the mark datasets of WordNet and Freebase. Experiment results tensor in NTN is set to 0.

9 Show significant and consistent improvements compared to Semantic Matching Energy (SME). SME model (Bor- state-of-the-art models. des et al. 2012; 2014) aims to capture correlations be- tween entities and relations via multiple matrix products and Related Models hadamard product . SME model simply represents each re- lation using a single vector, which interacts with Entity vec- TransE and TransH tors via linear matrix products, with all relations share the As mentioned in Section Introduction , TransE (Bordes et same parameters. SME considers two definitions of seman- al. 2013) wants h+r t when (h, r, t) holds. This indicates tic matching energy functions for optimization, including that (t) should be the nearest neighbor of (h + r). Hence, the linear form TransE assumes the score function fr (h, t) = (M1 h + M2 r + b1 )> (M3 t + M4 r + b2 ), (4).

10 Fr (h, t) = kh + r tk22 (1). and the bilinear form is low if (h, r, t) holds, and high otherwise. > . TransE applies well to 1-to-1 relations but has issues for fr (h, t) = (M1 h) (M2 r)+b1 (M3 t) (M4 r)+b2 , N-to-1, 1-to-N and N-to-N relations. Take a 1-to-N Relation r (5). for example. i {0, .. , m}, (hi , r, t) S. This indicates where M1 , M2 , M3 and M4 are weight matrices, is the that h0 = .. = hm , which does not comport with the facts. hadamard product , b1 and b2 are bias vectors. In (Bordes et To address the issue of TransE when modeling N-to-1, al. 2014), the bilinear form of SME is re-defined with 3-way 1-to-N and N-to-N relations, TransH (Wang et al. 2014) is tensors instead of matrices. proposed to enable an Entity to have distinct distributed rep- Latent Factor Model (LFM). LFM model (Jenatton et resentations when involved in different relations.)