ACTION-Net: Multipath Excitation for Action Recognition

1 Action -Net: Multipath Excitation for Action RecognitionZhengwei Wang1Qi She2 Aljosa Smolic11V-SENSE, Trinity College Dublin, Ireland2 ByteDance AI Lab, channel-wise, and motion patterns arethree complementary and crucial types of information forvideo Action Recognition . Conventional 2D CNNs are com-putationally cheap but cannot catch temporal relationships;3D CNNs can achieve good performance but are computa-tionally intensive. In this work, we tackle this dilemma bydesigning a generic and effective module that can be em-bedded into 2D CNNs. To this end, we propose a spAtio-temporal,Channel and moTion Excitation ( Action )module consisting of three paths: Spatio-Temporal Excita-tion (STE) path, Channel Excitation (CE) path, and MotionExcitation (ME) path.

2 The STE path employs one channel3D convolution to characterize spatio-temporal represen-tation. The CE path adaptively recalibrates channel-wisefeature responses by explicitly modeling interdependenciesbetween channels in terms of the temporal aspect. The MEpath calculates feature-level temporal differences, which isthen utilized to excite motion-sensitive channels. We equip2D CNNs with the proposed Action module to form a sim-ple yet effective Action -Net with very limited extra com-putational cost. Action -Net is demonstrated by consis-tently outperforming 2D CNN counterparts on three back-bones ( , ResNet-50, MobileNet V2 and BNInception)employing three datasets ( , Something-Something V2,Jester, and EgoGesture).

3 Codes are available IntroductionVideo understanding has attracted an increasing amountof interest, since it is a crucial step towards real-worldapplications, such as Virtual Reality/Augmented Reality(VR/AR) and video-sharing social networking instance, millions of videos are uploaded to Tik-Tok, Douyin, and Xigua Video to be processed everyday,This work is financially supported by Science Foundation Ireland(SFI) under the Grant Number 15/RP/2776. We gratefully acknowledgethe support of NVIDIA Corporation with the donation of the NVIDIADGX station used for this video understanding acts a pivotal part. However,the explosive growth in this video streaming gives rise tochallenges on performing video understanding at high ac-curacy and low computation Recognition , a fundamental problem in video un-derstanding, has been a growing demand in video-relatedapplications, such as content moderation ( , recognizecontent in videos that break terms of service) and contentrecommendations ( , videos are ranked by most liked andrecommended to similar customers).

4 The complex actionsin videos are normally temporal-dependent, which do notonly contain spatial information within each frame but alsoinclude temporal information over a duration. For example,symmetric Action pairs( opening a box , closing a box ),( rotate fists clockwise , rotate fists counterclockwise )contain similar features in spatial domains, but the tem-poral information is completely reversed. Traditional hu-man Action Recognition is morescene-related[36, 19, 16],wherein actions are not as temporal-dependent , applyeye makeup , walking , running . With how rapid technol-ogy is developing, like VR, which requires employing ges-tures to interact with environments,temporal-relatedactionrecog nition has recently become a focus for mainstreams of existing methods are 3D CNN-based frameworks and 2D CNN-based frameworks.

5 3 DCNNs have been shown to be effective in terms of spatio-temporal modeling [39, 4, 37], but spatio-temporal mod-eling is unable to capture adequate information containedin videos. The two-stream architecture was proposed totake spatio-temporal information and optical flow into ac-count [35, 3, 33], which boosted performance significantlycompared to the one-stream architecture. However, compu-tation on optical flow is very expensive, which poses chal-lenges on real-world applications. 3D CNNs suffer fromproblems including overfitting and slow convergence [8].With more large-scale datasets being released, such as Ki-netics [3], Moments in Time [29] and ActivityNet [2], op-timizing 3D CNNs becomes much easier and more popu-lar.

6 However, heavy computations inherent in 3D CNN-based frameworks contribute to slow inferences, whichwould limit their deployment on real-world applications,1 [ ] 11 Mar 2021 RawTSNTSMOursFigure 1:Visualization for significant features extracted by TSN, TSM and our Action -Net for the Action Rotate fists counterclockwise .Features extracted by each method are visualized by using CAM [48]. Compared to TSN and TSM, it can be noticed that Action -Netis able to extract features that are related to movements in an Action especially for highlighted frames , 4th and 5th columns. Moreexamples can be referred toSupplementary as VR that relies on online video Recognition .

7 Cur-rent 2D CNN-based frameworks [15, 35, 41, 47, 22] en-joy lightweight and fast inferences. These approaches op-erated on a sequence of short snippets (known as segments)sparsely sampled from the entire video and were initially in-troduced in TSN [41]. Original 2D CNNs lack the ability oftemporal modeling, which causes losing essential sequen-tial information in some actions , opening a box vs closing a box . TSM [22] introduced temporal informationto 2D CNN-based frameworks by shifting a part of chan-nels on the temporal axis, which significantly improved thebaseline for 2D CNN-based frameworks. However, TSMstill lacks explicit temporal modeling for an Action , suchas motion information.

8 Recent works [20, 14, 21, 24, 25]introduced embedded modules into 2D CNNs in terms ofResNet architecture [9], which possessed the capability formotion modeling. In order to capture multi-type informa-tion contained by videos, previous works normally oper-ated on input-level frames. For instance, SlowFast net-works sampled raw videos at multiple rates to character-ize slow and fast actions; two-stream networks utilized pre-computed optical flow for reasoning motion kind of approaches commonly require multi-branchnetworks, which need expensive by the aforementioned observation, we pro-pose a newplug-and-playandlightweightspAtio-tem poral,Channel and moTion Excitation ( Action ) module to ef-fectively process the multi-type information on the featurelevel inside a single network by adopting Multipath exci-tation.

9 The combination of spatio-temporal features andmotion features can be understood similarly as the two-stream architecture [35], but we model the motion insidethe network based on the feature level rather than generat-ing another type of input ( , optical flow [12]) for train-ing the network, which significantly reduces by SENet, the channel-wise features are extractedbased on the temporal domain to characterize the channelinterdependencies for the network. Correspondingly, a neu-ral architecture equipped with such a module is dubbedACTION-Net. Action comprises three components forextracting aforementioned features (1) Spatio-Temporal Ex-citation (STE), (2) Channel Excitation (CE) and (3) MotionExcitation (ME).

10 Figure 1 visualizes features characterizedby TSN, TSM, and Action -Net for the Action rotate fistscounterclockwise . It can be observed that both TSN andTSM focus on recognizing objects (two fists) independentlyinstead of reasoning an Action . Compared to TSN and TSM,our proposed Action -Net better characterizes an actionby representing feature maps that cover the two fists, espe-cially for the highlighted 4th and 5th columns. In a nutshell,our contributions are three-fold: We propose an Action module that works in aplug-and-playmanner, which is able to extract appropriatespatio-temporal patterns, channel-wise features, and2motion information to recognize actions.