Combining Multiple Sources of Knowledge in ... - …

1 Combining Multiple Sources of Knowledge in Deep CNNs for Action RecognitionEunbyung Park, Xufeng Han, tamara L. Berg, Alexander C. BergUniversity of North Carolina at Chapel deep convolutional neural networks (CNNs)have shown remarkable results for feature learning andprediction tasks, many recent studies have demonstratedimproved performance by incorporating additional hand-crafted features or by fusing predictions from multipleCNNs. Usually, these combinations are implemented viafeature concatenation or by averaging output predictionscores from several CNNs. In this paper, we present newapproaches for Combining different Sources of knowledgein deep learning. First, we propose feature amplification,where we use an auxiliary, hand-crafted, feature ( opti-cal flow) to perform spatially varying soft-gating on inter-mediate CNN feature maps.

2 Second, we present a spatiallyvarying multiplicative fusion method for Combining multi-ple CNNs trained on different Sources that results in robustprediction by amplifying or suppressing the feature activa-tions based on their agreement. We test these methods in thecontext of action recognition where information from spa-tial and temporal cues is useful, obtaining results that arecomparable with state-of-the-art methods and outperformmethods using only CNNs and optical flow IntroductionAs deep convolutional neural networks (CNNs) haveshown remarkable performance for computer vision taskson static images[11, 2, 15], there has been interest in ap-plying deep CNNs to video generally and to human actionrecognition in particular. However, for action recognition,the discriminative performance of spatio-temporal featureslearned by deep CNNs has fallen short compared to accu-racy gains seen in the image domain[10, 24].

3 There are twopotential reasons for this: 1) existing datasets are not largeor representative enough to learn robust features, and 2) itis difficult to either design or train networks to learn spatialand temporal features simultaneously. In order to overcomethese challenges, recent studies have proposed methods toincorporate hand-crafted features in combination with tra-ditional CNN based approaches that operate directly on rawpixels from video frames[18, 28, 13, 27, 25].Notable progress was made by Simonyan et al[18] whoproposed a two-stream network architecture designed tomimic the pathways of the human visual cortex for objectrecognition and motion recognition. One stream of theirapproach used a network focused on learning spatial fea-ture from RGB input images (spatial network), while theother stream used a network focused on learning temporalfeatures from optical flow inputs (temporal network).

4 Finalpredictions were computed as an average of the outputs ofthese two networks, demonstrating improved performanceover single stream approaches. Results indicated that errorsmade by the two networks were tuned to the particular inputfeature type, with the temporal network confusing differentclasses with similar motion patterns and the spatial networkconfusing different classes with the same depicted objects( human s face).Given these results, we believe that the two-stream ap-proach is promising, but could be strengthened by enhanc-ing each network with Knowledge from the other the spatial network, it could be useful to know whichparts of the image are moving. For example, the spatialnetwork could be improved by knowing that the hands andhammers are moving (in a hammering activity) so that thenetwork could focus on these portions of the frame ratherthan other non-essential parts of the frame.

5 On the otherhand, for the temporal network, it would be useful to haveknowledge about what is depicted in the moving parts ofa video. For example, if the temporal network knew thatthe repetitive up-down motion was a hammer, it could morereliably label the video as hammering. Therefore, ratherthan simply averaging the final outputs from the two net-works, we propose to incorporate this Knowledge directlyinto the networks to allow for earlier cross-communicationof Knowledge between the two this paper, we present two ways of Combining spatialand temporal cues in deep architectures for action recogni-tion. First, we propose feature amplification, where we usean auxiliary, hand-crafted, feature ( optical flow) to per-form spatially varying soft-gating on an intermediate CNNfeature map. In particular, we amplify the activations of thelast convolutional layer of a deep network by the magni-tude of optical flow, allowing a previously spatial-only net-work to understand which parts of image are moving be-fore the fully-connected layers are evaluated.

6 Second, wepresent a spatially varying multiplicative fusion method forcombining spatial feature maps ( the last convolutionallayers) from Multiple CNNs trained on different method amplifies or suppresses activations based onthe agreement between the networks in each part of theimage. Experimental results on two popular action recog-nition datasets, UCF101[21] and HMDB51[12], show thatthis outperforms previous additive fusion Related WorkThere have been many approaches for applying deeplearning methods to video classification. 3D CNN was pro-posed to train networks to learn spatio-temporal features[7,24]. [10] collected a large scale You-tube video datasetand applied several variants of CNN architectures, includ-ing early fusion, late fusion, and 3D CNN. [18] proposeda two-stream network to decouple spatial and temporal fea-tures.

7 Some fusion methods based on fully connected lay-ers have been suggested to improve video classification[9,26]. Recently, efforts to learn long-term temporal featureshave been proposed to train recurrent neural networks ( [6]) [1, 26] using hand-crafted features (as opposedto raw pixels) as inputs[25, 27].Multiplicative interactions have been explored in sev-eral places. [23] introduced a bilinear model that has mul-tiplicative interaction between two factors, such as con-tent and styles. [20] proposed a point-wise gated boltz-mann machine (PGBM) that can distinguish task-relevantfeatures from distracting task-irrelevant features by intro-ducing switching units that have multiplicative interactionsbetween visible and hidden units. [22] introduced a mul-tiplicative RNN which considers the multiplicative interac-tion between the current input and previous outputs of hid-den units.

8 Additionally, many variants of gated RNNs, suchas LSTM, have introduced multiplicative gates to controlthe amount of information that can be passed to CNNs were proposed very recently [14], usingtwo CNNs pre-trained on the ImageNet dataset and fine-tuned on fine-grained image classification datasets. Theircombination method extracts features from the last convo-lutional layer of each network and performs a dot-productbetween all possible pair-wise interactions to get a featurevector. Then they add one final fully-connected layer ontop of this for classification. This means that their modelis a linear combination of all possible pair-wise interactionsbetween feature maps. In terms of model capacity, theirmethod is similar to our multiplicative fusion method. How-ever, their method is less scalable because the size of theCNNE lement-wiseProductLast convfeaturesOptical flow magnitudeFully connected layers for classificationAmplifiedconvfeaturesRGB imageFigure 1: Feature amplification with optical flow magnitudefeature vectors are exponential with respect to the numberof feature maps, whereas ours does not depend on the sizeof feature maps and the number of networks.

9 Additionally,their networks are both trained for the same object classi-fication task, whereas our two-stream approach can incor-porate two explicitly different types of features, , spatialand temporal, into the classification Feature AmplificationCNNs have been successful for feature learning and se-lection that directly affect the discriminative power of clas-sifiers for many computer vision tasks. Especially for theimage classification task, where CNNs have outperformedhand-crafted features by a significant margin, extremelygood performance has been achieved using only categori-cal labels for images, leaving the CNNs to learn everythingelse such as objects, parts of objects, background, etc. It iswidely accepted that large-scale datasets and high computa-tional power have played an important role in these achieve-ments.

10 Recent work has shown that CNNs can even recog-nize and take advantage of characters on signs of store forstore front classification in large scale street view datasetswithout any supervision about where the text is located orwhat type of characters are depicted[16].However, collecting large-scale labeled datasets is veryexpensive. In some cases, such as the video domain, the sit-uation is even more challenging. However, without a largeenough dataset, CNNs may fail to learn and extract use-ful features since they could be easily distracted by noisyand unimportant features. They may also overfit to noisyand small training datasets, making generalization to thereal world infeasible. For example, for action recognitiondatasets in the video domain, such as UCF101, we foundout that existing spatial networks can easily be trained toachieve 100% accuracy on training dataset, but with rela-tively lower accuracy on testing dataset.