Multi-modal Knowledge Graphs for Recommender Systems

1 Multi-modal Knowledge Graphs for Recommender SystemsRui Sun1 , Xuezhi Cao2, Yan Zhao3, Junchen Wan2, Kun Zhou4, Fuzheng Zhang2 Zhongyuan Wang2and Kai Zheng1 1 School of Computer Science and Engineering, University of Electronic Science and Technology of China2 Meituan-Dianping Group3 Aalborg University, Danmark4 School of Information, Renmin University of {caoxuezhi, zhangfuzheng, Systems have shown great potential to solve theinformation explosion problem and enhance user experience invarious online applications. To tackle data sparsity and cold startproblems in Recommender Systems , researchers propose knowledgegraphs (KGs) based recommendations by leveraging valuable exter-nal Knowledge as auxiliary information.}

2 However, most of theseworks ignore the variety of data types ( , texts and images) inmulti- modal Knowledge Graphs (MMKGs). In this paper, we proposeMulti- modal Knowledge Graph Attention Network (MKGAT) tobetter enhance Recommender Systems by leveraging multi -modalknowledge. Specifically, we propose a Multi-modal graph attentiontechnique to conduct information propagation over MMKGs, andthen use the resulting aggregated embedding representation forrecommendation. To the best of our Knowledge , this is the firstwork that incorporates Multi-modal Knowledge graph into recom-mender Systems . We conduct extensive experiments on two realdatasets from different domains, results of which demonstrate thatour model MKGAT can successfully employ MMKGs to improvethe quality of recommendation CONCEPTS Information Systems Recommender Systems ;KEYWORDSR ecommender Systems , Graph Convolutional Networks, Multi-modal Knowledge GraphACM Reference Format:Rui Sun1 , Xuezhi Cao2, Yan Zhao3, Junchen Wan2, Kun Zhou4, FuzhengZhang2and Zhongyuan Wang2and Kai Zheng1.

3 2020. Multi-modal Knowl-edge Graphs for Recommender Systems . InProceedings of the 29th ACMI nternational Conference on Information and Knowledge Management (CIKM Corresponding author. This paper was done during the internship in Meituan-Dianping to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee.)

4 Request permissions from 20, October 19 23, 2020, Virtual Event, Ireland 2020 Association for Computing ISBN 978-1-4503-6859-9/20/10.. $ 20), October 19 23, 2020, Virtual Event, , New York, NY, USA,10 pages. INTRODUCTIONR ecently, Knowledge Graphs (KGs) are widely used in recommendersystems ( , KG-based recommendation) due to their comprehen-sive auxiliary data for effective recommendation [24,28]. Specifi-cally, the KG-based recommendation alleviates the sparsity problemof user-item interactions and the cold start problem by introducinghigh quality side information (KGs). These problems often arise inCollaborative Filtering (CF) [11] based , existing KG-based recommendation methods largelyignore the Multi-modal information, such as images and text de-scriptions of items.

5 Those visual or textual features may play asignificant role in recommendation Systems . For instance, beforewatching a movie, users tend to watch the trailer or read somerelated film reviews. When going to a restaurant for dinner, usersnormally browse the pictures of dishes or the reviews of the restau-rant on some online platforms, such as Yelp1or Dianping2atfirst. So it is necessary to introduce these Multi-modal informationinto Knowledge graph. The benefit is that Multi-modal knowledgegraphs (MKGs) introduce visual or textual information into theknowledge graph, regarding image or text as an entity or as anattribute of the entity. It is a more general way of acquiring externalmulti- modal Knowledge , without giving the expert definitions ofvisual or textual information.

6 A simple example of MKGs is shownin the figure Knowledge graph representation learning plays a key rolefor the KG-based recommendation. The KG-based recommendationmodels usually use the Knowledge graph representation model tolearn the embedding of the KGs entities, which are then fed into thedownstream recommendation task. There are two types of Multi-modal Knowledge graph representation learning: the feature-basedmethods and the entity-based feature-based methods [17,30] treat the modal informationas an auxiliary feature of the entity. It extends the translationalmodels (TransE) [2] by considering visual representations, whichare extracted from images corresponding to the Knowledge graphentities. The energy of a triple ( , the scoring function for triplesin TransE) is defined in terms of the structure of the KGs as well1 Storyis about the secret life oftoyswhen people are not StoryTom HanksWalt DisneyJohn Lasseterdirectoractorproducerhasimagehas descriptionFigure 1: Example of a Multi-modal Knowledge the visual representation of the entities.

7 However, the feature-based methods pose relatively requirements on the data sourceof the Knowledge graph since it requires that every entity in theknowledge graph has Multi-modal order to address the strict requirement on KGs data source,the entity-based methods [19] is proposed. The entity-based meth-ods treat different types of information ( , texts and images) asrelational triples of the structured Knowledge instead of auxiliaryfeatures, , first-class citizens of the Knowledge graph. It intro-duces visual and textual information by considering new relation,such as (denoting if an entity has image information)and (denoting if an entity has text information todescribe it). Then, the entity-based method processes each triple,( , , ), by independently applying translational models [2] or Con-volutional Neural Network (CNN) based models [18] to learn theknowledge graph embedding, where and denote a head andtail entity respectively, is the relationship ( , and ) between and.

8 Although the entity-based methods solve the problem of high de-mand for data sources of MKGs in the feature-based methods, it onlyfocuses on the reasoning relation between entities and ignores themulti- modal information fusion. In fact, Multi-modal information isusually used as an auxiliary information to enrich the informationof other entities. Therefore, we need a direct interactive way toexplicitly fuses the Multi-modal information into its correspondingentity before modeling the reasoning relation between the the limitations of the existing solutions, we believeit is essential to develop a MKGs representation model that canexploit MKGs in an efficient manner. Specifically, the model shouldsatisfy two conditions: 1) low requirements for MKGs data sources,2) Multi-modal information fusion is considered while preservingthe reasoning relation between entities.

9 Towards this end, we followthe entity-based methods to construct the Multi-modal knowledgegraph. And then, we propose Multi-modal Knowledge Graph Atten-tion Network (MKGAT), which models the Multi-modal knowledgegraph from two aspects: 1) entity information aggregation, whichaggregates the entity s neighbor node information to enrich theentity itself, 2) entity relation reasoning, which constructs reason-ing relations by the scoring function of the triple ( , TransE). Wefirst propose a new method to improve the graph attention neuralnetwork (GATs), which aggregates neighbor entities while takinginto account the relation in the Knowledge graph to complete entityinformation aggregation. And then we use a translational model tomodel the reasoning relation between entities.

10 A visible advantageof our MKGAT model lies in that it does not require each entityin the Knowledge graph to have Multi-modal information, whichmeans it has no particularly high requirements for Knowledge graphdata. Besides, the MKGAT model does not process each knowledgegraph triple independently but aggregates the neighbor informa-tion of the entity. As a result, it can learn the entity embedding thatfuses other modal information better. The primary contributions ofthis work can be summarized as follows: To the best of our Knowledge , this is the first work to intro-duce a Multi-modal Knowledge graph into a recommendationsystem. We develop a new MKGAT model, which employs informa-tion propagation on the Multi-modal Knowledge graph, toobtain better entity embedding for recommendation.