Heterogeneous Graph Neural Network

1 Heterogeneous Graph Neural NetworkChuxu ZhangUniversity of Notre SongNEC Laboratories America, HuangUniversity of Notre Dame, JD SwamiUS Army Research V. ChawlaUniversity of Notre learning in Heterogeneous graphs aims to pursuea meaningful vector representation for each node so as to facili-tate downstream applications such as link prediction, personalizedrecommendation, node classification,etc. This task, however, ischallenging not only because of the demand to incorporate het-erogeneous structural ( Graph ) information consisting of multipletypes of nodes and edges, but also due to the need for consideringheterogeneous attributes or contents (e.д., text or image) associ-ated with each node. Despite a substantial amount of effort hasbeen made to homogeneous (or Heterogeneous ) Graph embedding ,attributed Graph embedding as well as Graph Neural networks, fewof them can jointly consider Heterogeneous structural ( Graph ) infor-mation as well as Heterogeneous contents information of each nodeeffectively.

2 In this paper, we propose HetGNN, a heterogeneousgraph Neural Network model, to resolve this issue. Specifically, wefirst introduce a random walk with restart strategy to sample afixed size of strongly correlated Heterogeneous neighbors for eachnode and group them based upon node types. Next, we design aneural Network architecture with two modules to aggregate featureinformation of those sampled neighboring nodes. The first moduleencodes deep feature interactions of Heterogeneous contents andgenerates content embedding for each node. The second moduleaggregates content (attribute) embeddings of different neighboringgroups (types) and further combines them by considering the im-pacts of different groups to obtain the ultimate node , we leverage a Graph context loss and a mini-batch gradientdescent procedure to train the model in an end-to-end manner.

3 Ex-tensive experiments on several datasets demonstrate that HetGNNcan outperform state-of-the-art baselines in various Graph miningtasks, , link prediction, recommendation, node classification &clustering and inductive node classification & graphs, Graph Neural networks, Graph embeddingACM Reference Format:Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and NiteshV. Chawla. 2019. Heterogeneous Graph Neural Network . InThe 25th ACMP ublication rights licensed to ACM. ACM acknowledges that this contribution wasauthored or co-authored by an employee, contractor or affiliate of the United Statesgovernment. As such, the Government retains a nonexclusive, royalty-free right topublish or reproduce this article, or to allow others to do so, for Government 19, August 4 8, 2019, Anchorage, AK, USA 2019 Copyright held by the owner/author(s).

4 Publication rights licensed to ISBN 978-1-4503-6201-6/19/08.. $ Conference on Knowledge Discovery and Data Mining (KDD 19),August 4 8, 2019, Anchorage, AK, , New York, NY, USA, 11 INTRODUCTIONH eterogeneous graphs (HetG) [26,27] contain abundant informa-tion with structural relations (edges) among multi-typed nodes aswell as unstructured content associated with each node. For in-stance, the academic Graph in Fig. 1(a) denotes relations betweenauthors and papers (write), papers and papers (cite), papers andvenues (publish),etc. Moreover, nodes in this Graph carry attributes(e.д., author id) and text (e.д., paper abstract). Another exampleillustrates user-item relations in the review Graph and nodes areassociated with attributes (e.д., user id), text (e.д.)

5 , item description)and image (e.д., item picture). This ubiquity of HetG has led to aninflux of research on corresponding Graph mining methods andalgorithms such as relation inference [2,25,33,35], personalizedrecommendation [10, 23], node classification [36], , a variety of these HetG tasks have relied on fea-ture vectors derived from a manual feature engineering tasks. Thisrequires specifications and computation of different statistics orproperties about the HetG as a feature vector for downstream ma-chine learning or analytic tasks. However, this can be very limitingand not generalizable. More recently, there has been an emergenceof representation learning approaches to automate the feature engi-neering tasks, which can then facilitate a multitude of downstreammachine learning or analytic tasks.

6 Beginning with homogeneousgraphs [6,20,29], Graph representation learning has been expandedto Heterogeneous graphs [1,4], attributed graphs [15,34] as wellas specific graphs [22,28]. For instance, the shallow models,e.д.,DeepWalk [20], were initially developed to feed a set of short ran-dom walks over the Graph to the SkipGram model [19] so as toapproximate the node co-occurrence probability in these walksand obtain node embeddings. Subsequently, semantic-aware ap-proaches,e.д., metapath2vec [4], were proposed to address nodeand relation heterogeneity in Heterogeneous graphs. In addition,content-aware approaches,e.д., ASNE [15], leveraged both latent features and attributes to learn node embeddings in the methods learn node latent embeddings directly, but arelimited in capturing the rich neighborhood information.

7 The GraphNeural Networks (GNNs) employ deep Neural networks to aggre-gate feature information of neighboring nodes, which makes theaggregated embedding more powerful. In addition, the GNNs canbe naturally applied to inductive tasks involving nodes that are notpresent in the training period. For instance, GCN [12], GraphSAGER esearch Track PaperKDD 19, August 4 8, 2019, Anchorage, AK, USA793attributestext.. image.. attributesimage.. (b)(a)paperauthorvenuea1a2a3a4v1v2p1p2p3 p4itemuseri1i2i3i4u1u2u3u4academicgraphr eviewgraphFigure 1: (a) HetG examples: an academic Graph and a reviewgraph. (b) Challenges of Graph Neural Network for HetG: C1- sampling Heterogeneous neighbors (for nodeain this case,node colors denote different types); C2 - encoding heteroge-neous contents; C3 - aggregating Heterogeneous neighbors.

8 [7], and GAT [31] employ convolutional operator, LSTM architec-ture, and self-attention mechanism to aggregate feature informationof neighboring nodes, respectively. The advances and applicationsof GNNs are largely concentrated on homogeneous graphs. Currentstate-of-the-art GNNs have not well solved the following challengesfaced for HetG, which we address in this paper. (C1) Many nodes in HetG may not connect to all types of neigh-bors. In addition, the number of neighboring nodes varies fromnode to node. For example, in Figure 1(a), any author node hasno direct connection to a venue node. Meanwhile, in Figure 1(b),nodeahas 5 direct neighbors while nodeconly has 2. Mostexisting GNNs only aggregate feature information of direct (first-order) neighboring nodes and the feature propagation processmay weaken the effect of farther neighbors.

9 Moreover, the embed-ding generation of hub node is impaired by weakly correlatedneighbors ( noise neighbors) and the embedding of cold-start node is not sufficiently represented due to limited neighbor infor-mation. Thus challenge 1 is:how to sample Heterogeneous neighborsthat are strongly correlated to embedding generation for each nodein HetG, as indicated by C1 in Figure 1(b)? (C2) A node in HetG can carry unstructured Heterogeneous con-tents,e.д., attributes, text or image. In addition, content associatedwith different types of nodes can be different. For example, inFigure 1(b), type-1 nodes (e.д.,borc) contain attributes and textcontent, type-2 nodes (e.д.,forд) carry attributes and image,type-k nodes (e.д.,dore) are associated with text and direct concatenation operation or linear transformation bythe current GNNs cannot model deep interactions among nodeheterogeneous contents.

10 Moreover, it is not applicable to use theTable 1: Model comparison: (1) RL - representation learning?(2) HG - Heterogeneous Graph ? (3) C - content aware? (4) HC- Heterogeneous contents aware? (5) I - inductive inference?PropertyDWMP2 VASNESHNEGSAGEGATHetGNN[20][4][15][34][7 ][31]RL HG C HC I same feature transformation function for all node types as theircontents vary from each other. Thus challenge 2 is:how to de-sign node content encoder for addressing content heterogeneity ofdifferent nodes in HetG, as indicated by C2 in Figure 1(b)? (C3) Different types of neighbors contribute differently to thenode embeddings in HetG. For example, in the academic graphof Figure 1(a), author and paper neighbors should have moreinfluence on the embedding of author node as a venue nodecontains diverse topics thus has more general embedding .