Abstract - arXiv.org e-Print archive

1 A Convolutional Neural Network for Modelling SentencesNal KalchbrennerEdward Grefenstette{ , , of Computer ScienceUniversity of OxfordPhil BlunsomAbstractThe ability to accurately represent sen-tences is central to language understand-ing. We describe a convolutional architec-ture dubbed the Dynamic ConvolutionalNeural Network (DCNN) that we adoptfor the semantic modelling of network uses Dynamick-Max Pool-ing, a global pooling operation over lin-ear sequences. The network handles inputsentences of varying length and inducesa feature graph over the sentence that iscapable of explicitly capturing short andlong-range relations.}

2 The network doesnot rely on a parse tree and is easily ap-plicable to any test theDCNN in four experiments: small scalebinary and multi-class sentiment predic-tion, six-way question classification andTwitter sentiment prediction by distant su-pervision. The network achieves excellentperformance in the first three tasks and agreater than25%error reduction in the lasttask with respect to the strongest IntroductionThe aim of a sentence model is to analyse andrepresent the semantic content of a sentence forpurposes of classification or generation.

3 The sen-tence modelling problem is at the core of manytasks involving a degree of natural language com-prehension. These tasks include sentiment analy-sis, paraphrase detection, entailment recognition,summarisation, discourse analysis, machine trans-lation, grounded language learning and image re-trieval. Since individual sentences are rarely ob-served or not observed at all, one must representa sentence in terms of features that depend on thewords and shortn-grams in the sentence that arefrequently observed.

4 The core of a sentence modelinvolves a feature function that defines the process The cat sat on the red mat The cat sat on the red matFigure 1: Subgraph of a feature graph inducedover an input sentence in a Dynamic Convolu-tional Neural Network. The full induced graphhas multiple subgraphs of this kind with a distinctset of edges; subgraphs may merge at differentlayers. The left diagram emphasises the poolednodes. The width of the convolutional filters is 3and 2 respectively.

5 With dynamic pooling, a fil-ter with small width at the higher layers can relatephrases far apart in the input which the features of the sentence are extractedfrom the features of the words types of models of meaning have beenproposed. Composition based methods have beenapplied to vector representations of word meaningobtained from co-occurrence statistics to obtainvectors for longer phrases. In some cases, com-position is defined by algebraic operations overword meaning vectors to produce sentence mean-ing vectors (Erk and Pad o, 2008; Mitchell andLapata, 2008; Mitchell and Lapata, 2010; Tur-ney, 2012; Erk, 2012; Clarke, 2012).

6 In othercases, a composition function is learned and ei-ther tied to particular syntactic relations (Guevara,2010; Zanzotto et al., 2010) or to particular wordtypes (Baroni and Zamparelli, 2010; Coecke etal., 2010; Grefenstette and Sadrzadeh, 2011; Kart-saklis and Sadrzadeh, 2013; Grefenstette, 2013).Another approach represents the meaning of sen-tences by way of automatically extracted logicalforms (Zettlemoyer and Collins, 2005). [ ] 8 Apr 2014A central class of models are those based onneural networks.

7 These range from basic neu-ral bag-of-words or bag-of-n-grams models to themore structured recursive neural networks andto time-delay neural networks based on convo-lutional operations (Collobert and Weston, 2008;Socher et al., 2011; Kalchbrenner and Blunsom,2013b).Neural sentence models have a num-ber of advantages. They can be trained to obtaingeneric vectors for words and phrases by predict-ing, for instance, the contexts in which the wordsand phrases occur. Through supervised training,neural sentence models can fine-tune these vec-tors to information that is specific to a certaintask.

8 Besides comprising powerful classifiers aspart of their architecture, neural sentence modelscan be used to condition a neural language modelto generate sentences word by word (Schwenk,2012; Mikolov and Zweig, 2012; Kalchbrennerand Blunsom, 2013a).We define a convolutional neural network archi-tecture and apply it to the semantic modelling ofsentences. The network handles input sequencesof varying length. The layers in the network in-terleave one-dimensional convolutional layers anddynamick-max pooling layers.

9 Dynamick-maxpooling is a generalisation of the max pooling op-erator. The max pooling operator is a non-linearsubsampling function that returns the maximumof a set of values (LeCun et al., 1998). The op-erator is generalised in two respects. First,k-max pooling over a linear sequence of values re-turns the subsequence ofkmaximum values in thesequence, instead of the single maximum , the pooling parameterkcan be dynam-ically chosen by makingka function of other as-pects of the network or the filters across each row of features inthe sentence matrix.

10 Convolving the same filterwith then-gram at every position in the sentenceallows the features to be extracted independentlyof their position in the sentence. A convolutionallayer followed by a dynamic pooling layer anda non-linearity form a feature map. Like in theconvolutional networks for object recognition(LeCun et al., 1998), we enrich the representationin the first layer by computing multiple featuremaps with different filters applied to the inputsentence.