Convolutional Neural Networks for Sentence Classification

1 Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746 1751,October 25-29, 2014, Doha, 2014 Association for Computational LinguisticsConvolutional Neural Networks for Sentence ClassificationYoon KimNew York report on a series of experiments withconvolutional Neural Networks (CNN)trained on top of pre-trained word vec-tors for Sentence -level Classification show that a simple CNN with lit-tle hyperparameter tuning and static vec-tors achieves excellent results on multi-ple task-specificvectors through fine-tuning offers furthergains in additionallypropose a simple modification to the ar-chitecture to allow for the use of bothtask-specific and static vectors.

2 The CNNmodels discussed herein improve upon thestate of the art on 4 out of 7 tasks, whichinclude sentiment analysis and IntroductionDeep learning models have achieved remarkableresults in computer vision (Krizhevsky et al.,2012) and speech recognition (Graves et al., 2013)in recent years. Within natural language process-ing, much of the work with deep learning meth-ods has involved learning word vector representa-tions through Neural language models (Bengio etal., 2003; Yih et al., 2011; Mikolov et al., 2013)and performing composition over the learned wordvectors for Classification (Collobert et al., 2011).Word vectors, wherein words are projected from asparse, 1-of-Vencoding (hereVis the vocabularysize) onto a lower dimensional vector space via ahidden layer, are essentially feature extractors thatencode semantic features of words in their dimen-sions.

3 In such dense representations, semanticallyclose words are likewise close in euclidean orcosine distance in the lower dimensional Neural Networks (CNN) utilizelayers with convolving filters that are applied tolocal features (LeCun et al., 1998). Originallyinvented for computer vision, CNN models havesubsequently been shown to be effective for NLPand have achieved excellent results in semanticparsing (Yih et al., 2014), search query retrieval(Shen et al., 2014), Sentence modeling (Kalch-brenner et al., 2014), and other traditional NLPtasks (Collobert et al., 2011).In the present work, we train a simple CNN withone layer of convolution on top of word vectorsobtained from an unsupervised Neural languagemodel.

4 These vectors were trained by Mikolov etal. (2013) on 100 billion words of Google News,and are publicly initially keep theword vectors static and learn only the other param-eters of the model. Despite little tuning of hyper-parameters, this simple model achieves excellentresults on multiple benchmarks, suggesting thatthe pre-trained vectors are universal feature ex-tractors that can be utilized for various classifica-tion tasks. Learning task-specific vectors throughfine-tuning results in further improvements. Wefinally describe a simple modification to the archi-tecture to allow for the use of both pre-trained andtask-specific vectors by having multiple work is philosophically similar to Razavianet al.

5 (2014) which showed that for image clas-sification, feature extractors obtained from a pre-trained deep learning model perform well on a va-riety of tasks including tasks that are very dif-ferent from the original task for which the featureextractors were ModelThe model architecture, shown in figure 1, is aslight variant of the CNN architecture of Collobertet al. (2011). Letxi Rkbe thek-dimensionalword vector corresponding to thei-th word in thesentence. A Sentence of lengthn(padded where1 for the video and do n't rent it n x k representation of Sentence with static and non-static channels Convolutional layer with multiple filter widths and feature maps Max-over-time pooling Fully connected layer with dropout and softmax output Figure 1: Model architecture with two channels for an example ) is represented asx1:n=x1 x2.

6 Xn,(1)where is the concatenation operator. In gen-eral, letxi:i+jrefer to the concatenation of wordsxi,xi+1, .. ,xi+j. A convolution operation in-volves afilterw Rhk, which is applied to awindow ofhwords to produce a new feature. Forexample, a featureciis generated from a windowof wordsxi:i+h 1byci=f(w xi:i+h 1+b).(2)Hereb Ris a bias term andfis a non-linearfunction such as the hyperbolic tangent. This filteris applied to each possible window of words in thesentence{x1:h,x2:h+1, .. ,xn h+1:n}to produceafeature mapc= [c1, c2, .. , cn h+1],(3)withc Rn h+1. We then apply a max-over-time pooling operation (Collobert et al., 2011)over the feature map and take the maximum value c= max{c}as the feature corresponding to thisparticular filter.

7 The idea is to capture the most im-portant feature one with the highest value foreach feature map. This pooling scheme naturallydeals with variable Sentence have described the process by whichonefeature is extracted fromonefilter. The modeluses multiple filters (with varying window sizes)to obtain multiple features. These features formthe penultimate layer and are passed to a fully con-nected softmax layer whose output is the probabil-ity distribution over one of the model variants, we experimentwith having two channels of word vectors onethat is kept static throughout training and one thatis fine-tuned via backpropagation (section ).2In the multichannel architecture, illustrated in fig-ure 1, each filter is applied to both channels andthe results are added to calculateciin equation(2).

8 The model is otherwise equivalent to the sin-gle channel RegularizationFor regularization we employ dropout on thepenultimate layer with a constraint onl2-norms ofthe weight vectors (Hinton et al., 2012). Dropoutprevents co-adaptation of hidden units by ran-domly dropping out , setting to zero a pro-portionpof the hidden units during foward-backpropagation. That is, given the penultimatelayerz= [ c1, .. , cm](note that here we havemfilters), instead of usingy=w z+b(4)for output unityin forward propagation, dropoutusesy=w (z r) +b,(5)where is the element-wise multiplication opera-tor andr Rmis a masking vector of Bernoullirandom variables with probabilitypof being are backpropagated only through theunmasked units.

9 At test time, the learned weightvectors are scaled bypsuch that w=pw, and wis used (without dropout) to score unseen sen-tences. We additionally constrainl2-norms of theweight vectors by rescalingwto have||w||2=swhenever||w||2> safter a gradient descent employ language from computer vision where a colorimage has red, green, and blue |V||Vpre|TestMR220106621876516448 CVSST-15181185517836162622210 SST-2219961316185148381821 Subj223100002132317913 CVTREC610595295929125500CR21937755340504 6 CVMPQA231060662466083 CVTable 1: Summary statistics for the datasets after : Number of target : Average Sentence : Dataset size.|V|: Vocabulary size.|Vpre|: Number ofwords present in the set of pre-trained word :Test set size (CV means there was no standard train/test splitand thus 10-fold CV was used).

10 3 Datasets and Experimental SetupWe test our model on various benchmarks. Sum-mary statistics of the datasets are in table 1. MR: Movie reviews with one Sentence per re-view. Classification involves detecting posi-tive/negative reviews (Pang and Lee, 2005).3 SST-1: Stanford Sentiment Treebank anextension of MR but with train/dev/test splitsprovided and fine-grained labels (very pos-itive, positive, neutral, negative, very nega-tive), re-labeled by Socher et al. (2013).4 SST-2: Same as SST-1 but with neutral re-views removed and binary labels. Subj: Subjectivity dataset where the task isto classify a Sentence as being subjective orobjective (Pang and Lee, 2004). TREC: TREC question dataset task in-volves classifying a question into 6 questiontypes (whether the question is about person,location, numeric information, etc.)