arXiv:2106.06090v1 [cs.CL] 10 Jun 2021

1 GRAPHNEURALNETWORKS FORNLP: A SURVEYG raph Neural Networks for Natural Language Processing:A SurveyLingfei Wu Silicon Valley Research Center, USAYu Chen Polytechnic Institute, USAKai Shen University, ChinaXiaojie Silicon Valley Research Center, USAH anning China Normal University, ChinaShucheng Li University, ChinaJian Fraser University, CanadaBo ChinaAbstractDeep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP). Although text inputs are typically represented as a sequence of tokens, there isa rich variety of NLP problems that can be best expressed with a graph structure .

2 As a result, thereis a surge of interests in developing new deep learning techniques on graphs for a large numberof NLP tasks. In this survey, we present a comprehensive overview onGraph Neural Networks(GNNs) for Natural Language Processing. We propose a new taxonomy of GNNs for NLP, whichsystematically organizes existing research of GNNs for NLP along three axes: graph construction, graph representation learning , and graph based encoder-decoder models. We further introducea large number of NLP applications that are exploiting the power of GNNs and summarize thecorresponding benchmark datasets, evaluation metrics, and open-source codes.

3 Finally, we discussvarious outstanding challenges for making the full use of GNNs for NLP as well as future researchdirections. To the best of our knowledge, this is the first comprehensive overview of graph NeuralNetworks for Natural Language : graph Neural Networks, Natural Language Processing, Deep learning on Graphs1. IntroductionDeep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP) today, especially when operated on large-scale text corpora. Conventionally, text*. Both authors contributed equally to this research.

4 This research is done when Kai Shen is an intern at . Shucheng Li is also with National Key Lab for Novel Software Technology, Nanjing [ ] 10 Jun 2021 LINGFEIWU ANDYUCHEN,ET are considered as a bag of tokens such as BoW and TF-IDF in NLP tasks. With recentsuccess of Word Embeddings techniques (Mikolov et al., 2013; Pennington et al., 2014), sentencesare typically represented as a sequence of tokens in NLP tasks. Hence, popular deep learningtechniques such as recurrent neural networks (Schuster and Paliwal, 1997) and convolutional neuralnetworks (Krizhevsky et al.)

5 , 2012) have been widely applied for modeling text , there is a rich variety of NLP problems that can be best expressed with a graph struc-ture. For instance, the sentence structural information in text sequence ( syntactic parsing treeslike dependency and constituency parsing trees) can be exploited to augment original sequence databy incorporating the task-specific knowledge. Similarly, the semantic information in sequence data( semantic parsing graphs like Abstract Meaning Representation graphs and Information Extrac-tion graphs) can be leveraged to enhance original sequence data as well.

6 Therefore, these graph -structured data can encode complicated pairwise relationships between entity tokens for learningmore informative , deep learning techniques that were disruptive for Euclidean data ( , images)or sequence data ( , text) are not immediately applicable to graph -structured data, due to thecomplexity of graph data such as irregular structure and varying size of node neighbors. As a result,this gap has driven a tide in research for deep learning on graphs, especially in development of graphneural networks (GNNs) (Kipf and Welling, 2016; Defferrard et al.)

7 , 2016; Hamilton et al., 2017a).This wave of research at the intersection of deep learning on graphs and NLP has influenceda variety of NLP tasks. There has seen a surge of interests in applying and developing differentGNNs variants and achieved considerable success in many NLP tasks, ranging from classificationtasks like sentence classification (Henaff et al., 2015; Huang and Carley, 2019), semantic role la-beling Luo and Zhao (2020); Gui et al. (2019), and relation extraction (Qu et al., 2020; Sahu et al.,2019), to generation tasks like machine translation (Bastings et al.

8 , 2017; Beck et al., 2018), ques-tion generation (Pan et al., 2020; Sachan et al., 2020), and summarization (Fernandes et al., 2019;Yasunaga et al., 2017). Despite the successes these existing research has achieved, deep learning ongraphs for NLP still encounters many challenges, namely, Automatically transforming original text sequence data into highly graph -structured challenge is profound in NLP since most of the NLP tasks involving using the textsequences as the original inputs. Automatic graph construction from the text sequence to uti-lize the underlying structural information is a crucial step in utilizing graph neural networksfor NLP problems.

9 Properly determining graph representation learning techniques. It is critical to come up withspecially-designed GNNs to learn the unique characteristics of different graph -structures datasuch as undirected, directed, multi-relational and heterogeneous graphs. Effectively modeling complex data. Such challenge is important since many NLP tasks in-volve learning the mapping between the graph -based inputs and other highly structured outputdata such as sequences, trees, as well as graph data with multi-types in both nodes and this survey, we will present for the first time a comprehensive overview ofGraph Neural Net-works for Natural Language Processing.

10 Our survey is timely for both Machine learning and NLPcommunities, which covers relevant and interesting topics, including automatic graph constructionfor NLP, graph representation learning for NLP, various advanced GNNs-based encoder-decoder2 GRAPHNEURALNETWORKS FORNLP: A SURVEYF igure 1: The taxonomy, which systematically organizes GNNs for NLP along three axes: graphconstruction, graph representation learning , encoder-decoder models, and the ( graph2seq, graph2tree, and graph2graph) for NLP, and the applications of GNNs invarious NLP tasks. We highlight our main contributions as follows: We propose a new taxonomy of GNNs for NLP, which systematically organizes existing re-search of GNNs for NLP along three axes: graph construction, graph representation learning ,and graph based encoder-decoder models.