ACTION-Net: Multipath Excitation for Action Recognition

1 Action -Net: Multipath Excitation for Action RecognitionZhengwei Wang1Qi She2 Aljosa Smolic11V-SENSE, Trinity College Dublin, Ireland2 ByteDance AI Lab, channel-wise, and motion patterns arethree complementary and crucial types of information forvideo Action Recognition . Conventional 2D CNNs are com-putationally cheap but cannot catch temporal relationships;3D CNNs can achieve good performance but are computa-tionally intensive. In this work, we tackle this dilemma bydesigning a generic and effective module that can be em-bedded into 2D CNNs. To this end, we propose a spAtio-temporal,Channel and moTion Excitation ( Action )module consisting of three paths: Spatio-Temporal Excita-tion (STE) path, Channel Excitation (CE) path, and MotionExcitation (ME) path. The STE path employs one channel3D convolution to characterize spatio-temporal represen-tation.

2 The CE path adaptively recalibrates channel-wisefeature responses by explicitly modeling interdependenciesbetween channels in terms of the temporal aspect. The MEpath calculates feature-level temporal differences, which isthen utilized to excite motion-sensitive channels. We equip2D CNNs with the proposed Action module to form a sim-ple yet effective Action -Net with very limited extra com-putational cost. Action -Net is demonstrated by consis-tently outperforming 2D CNN counterparts on three back-bones ( , ResNet-50, MobileNet V2 and BNInception)employing three datasets ( , Something-Something V2,Jester, and EgoGesture). Codes are available IntroductionVideo understanding has attracted an increasing amountof interest, since it is a crucial step towards real-worldapplications, such as Virtual Reality/Augmented Reality(VR/AR) and video-sharing social networking instance, millions of videos are uploaded to Tik-Tok, Douyin, and Xigua Video to be processed everyday,This work is financially supported by Science Foundation Ireland(SFI) under the Grant Number 15/RP/2776.

3 We gratefully acknowledgethe support of NVIDIA Corporation with the donation of the NVIDIADGX station used for this video understanding acts a pivotal part. However,the explosive growth in this video streaming gives rise tochallenges on performing video understanding at high ac-curacy and low computation Recognition , a fundamental problem in video un-derstanding, has been a growing demand in video-relatedapplications, such as content moderation ( , recognizecontent in videos that break terms of service) and contentrecommendations ( , videos are ranked by most liked andrecommended to similar customers). The complex actionsin videos are normally temporal-dependent, which do notonly contain spatial information within each frame but alsoinclude temporal information over a duration. For example,symmetric Action pairs( opening a box , closing a box ),( rotate fists clockwise , rotate fists counterclockwise )contain similar features in spatial domains, but the tem-poral information is completely reversed.

4 Traditional hu-man Action Recognition is morescene-related[36, 19, 16],wherein actions are not as temporal-dependent , applyeye makeup , walking , running . With how rapid technol-ogy is developing, like VR, which requires employing ges-tures to interact with environments,temporal-relatedactionrecog nition has recently become a focus for mainstreams of existing methods are 3D CNN-based frameworks and 2D CNN-based frameworks. 3 DCNNs have been shown to be effective in terms of spatio-temporal modeling [39, 4, 37], but spatio-temporal mod-eling is unable to capture adequate information containedin videos. The two-stream architecture was proposed totake spatio-temporal information and optical flow into ac-count [35, 3, 33], which boosted performance significantlycompared to the one-stream architecture.

5 However, compu-tation on optical flow is very expensive, which poses chal-lenges on real-world applications. 3D CNNs suffer fromproblems including overfitting and slow convergence [8].With more large-scale datasets being released, such as Ki-netics [3], Moments in Time [29] and ActivityNet [2], op-timizing 3D CNNs becomes much easier and more popu-lar. However, heavy computations inherent in 3D CNN-based frameworks contribute to slow inferences, whichwould limit their deployment on real-world applications,1 [ ] 11 Mar 2021 RawTSNTSMOursFigure 1:Visualization for significant features extracted by TSN, TSM and our Action -Net for the Action Rotate fists counterclockwise .Features extracted by each method are visualized by using CAM [48]. Compared to TSN and TSM, it can be noticed that Action -Netis able to extract features that are related to movements in an Action especially for highlighted frames , 4th and 5th columns.

6 Moreexamples can be referred toSupplementary as VR that relies on online video Recognition . Cur-rent 2D CNN-based frameworks [15, 35, 41, 47, 22] en-joy lightweight and fast inferences. These approaches op-erated on a sequence of short snippets (known as segments)sparsely sampled from the entire video and were initially in-troduced in TSN [41]. Original 2D CNNs lack the ability oftemporal modeling, which causes losing essential sequen-tial information in some actions , opening a box vs closing a box . TSM [22] introduced temporal informationto 2D CNN-based frameworks by shifting a part of chan-nels on the temporal axis, which significantly improved thebaseline for 2D CNN-based frameworks. However, TSMstill lacks explicit temporal modeling for an Action , suchas motion information. Recent works [20, 14, 21, 24, 25]introduced embedded modules into 2D CNNs in terms ofResNet architecture [9], which possessed the capability formotion modeling.

7 In order to capture multi-type informa-tion contained by videos, previous works normally oper-ated on input-level frames. For instance, SlowFast net-works sampled raw videos at multiple rates to character-ize slow and fast actions; two-stream networks utilized pre-computed optical flow for reasoning motion kind of approaches commonly require multi-branchnetworks, which need expensive by the aforementioned observation, we pro-pose a newplug-and-playandlightweightspAtio-tem poral,Channel and moTion Excitation ( Action ) module to ef-fectively process the multi-type information on the featurelevel inside a single network by adopting Multipath exci-tation. The combination of spatio-temporal features andmotion features can be understood similarly as the two-stream architecture [35], but we model the motion insidethe network based on the feature level rather than generat-ing another type of input ( , optical flow [12]) for train-ing the network, which significantly reduces by SENet, the channel-wise features are extractedbased on the temporal domain to characterize the channelinterdependencies for the network.

8 Correspondingly, a neu-ral architecture equipped with such a module is dubbedACTION-Net. Action comprises three components forextracting aforementioned features (1) Spatio-Temporal Ex-citation (STE), (2) Channel Excitation (CE) and (3) MotionExcitation (ME). Figure 1 visualizes features characterizedby TSN, TSM, and Action -Net for the Action rotate fistscounterclockwise . It can be observed that both TSN andTSM focus on recognizing objects (two fists) independentlyinstead of reasoning an Action . Compared to TSN and TSM,our proposed Action -Net better characterizes an actionby representing feature maps that cover the two fists, espe-cially for the highlighted 4th and 5th columns. In a nutshell,our contributions are three-fold: We propose an Action module that works in aplug-and-playmanner, which is able to extract appropriatespatio-temporal patterns, channel-wise features, and2motion information to recognize actions.

9 A simple yet effective neural architecture referred toACTION-Net, which we demonstrate on three back-bones , ResNet-50 [9], BNIception [13] and Mo-bileNet V2 [31]. We have conducted extensive experiments andshown our superior performances on three datasetsSomething-Something V2 [7], Jester [27] and EgoGes-ture [46].2. Related WorksIn this section, we discuss related works by taking2D and 3D CNN-based frameworks into account, whereinSENet [11] and TEA [21] inspired us to propose 3D CNN-based FrameworkThe 3D CNN-based framework has a spatio-temporalmodeling capability, which enhances model performancefor video Action Recognition [39, 8, 39]. I3D [3] inflated theImageNet pre-trained 2D kernels to 3D kernels for captur-ing spatio-temporal information. To better represent motionpatterns, I3D utilized pre-computed optical flow togetherwith RGB (also known as the two-stream architecture).

10 SlowFast networks [5] were proposed to handle inconsis-tent speeds of actions in videos , running and walking,which involved a slow branch and a fast branch to modelslow actions and fast actions, respectively. Although 3 DCNN-based approaches have achieved exciting results onseveral benchmark datasets, they contain massive parame-ters. In this case, various problems are caused, such as eas-ily overfitting [8] and difficulty in converging [40], whichpose challenges including ineffective inferences for onlinestreaming videos in real-world applications. Although re-cent works [30, 40] have demonstrated that 3D convolutioncan be factorized to lessen computations to some extent, thecomputation is still much more of a burden when comparedto 2D CNN-based 2D CNN-based FrameworkTSN [41] was the first proposed framework that ap-plied 2D CNNs for video Action Recognition , which intro-duced the concept of segment to process videos , ex-tract short snippets over a long video sequence with a uni-form sparse sampling scheme.