1 Squeeze-and-Excitation Networks

1 1 Squeeze-and-Excitation NetworksJie Hu[0000 0002 5150 1003]Li Shen[0000 0002 2283 4976]Samuel Albanie[0000 0001 9736 5134]Gang Sun[0000 0001 6913 6799]Enhua Wu[0000 0002 2174 1428]Abstract The central building block of convolutional neural Networks (CNNs) is the convolution operator, which enables Networks toconstruct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broadrange of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power ofa CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channelrelationship and propose a novel architectural unit, which we term the Squeeze-and-Excitation (SE) block, that adaptively recalibrateschannel-wise feature responses by explicitly modelling interdependencies between channels.

2 We show that these blocks can bestacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstratethat SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at minimal additional computationalcost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC2017classification submission which won first place andreduced the top-5error , surpassing the winning entry of2016by a relative improvement of 25%. Models and code areavailable at Terms Squeeze-and-Excitation , Image classification, Convolutional Neural INTRODUCTIONCONVOLUTIONAL neural Networks (CNNs) have provento be useful models for tackling a wide range ofvisual tasks [1] [4]. At each convolutional layer in thenetwork, a collection of filters expresses neighbourhoodspatial connectivity patterns along input channels fusingspatial and channel-wise information together within localreceptive fields.

3 By interleaving a series of convolutionallayers with non-linear activation functions and downsam-pling operators, CNNs are able to produce robust represen-tations that capture hierarchical patterns and attain globaltheoretical receptive fields. Recent research has shown thatthese representations can be strengthened by integratinglearning mechanisms into the network that help capturespatial correlations between features. One such approach,popularised by the Inception family of architectures [5], [6],incorporates multi-scale processes into network modules toachieve improved performance. Further work has sought tobetter model spatial dependencies [7], [8] and incorporatespatial attention into the structure of the network [9].

4 In this paper, we investigate a different aspect of networkdesign - the relationship between channels. We introducea new architectural unit, which we term theSqueeze-and- excitation (SE) block, with the goal of improving the quality Jie Hu and Enhua Wu are with the State Key Laboratory of ComputerScience, Institute of Software, Chinese Academy of Sciences, Beijing,100190, are also with the University of Chinese Academy of Sciences, Beijing,100049, Hu is also with Momenta and Enhua Wu is also with the Universityof Gang Sun is with LIAMA-NLPR at the CAS Institute of Automation Li Shen and Samuel Albanie are with the Visual Geometry Group at theUniversity of representations produced by a network by explicitlymodelling the interdependencies between the channels ofits convolutional features.

5 To this end, we propose a mecha-nism that allows the network to performfeature recalibration,through which it can learn to use global information toselectively emphasise informative features and suppress lessuseful structure of the SE building block is depicted inFig. 1. For any given transformationFtr:X7 U,X RH W C ,U RH W C, ( a convolution), wecan construct a corresponding SE block to perform featurerecalibration. The featuresUare first passed through asqueezeoperation, which produces a channel descriptor byaggregating feature maps across their spatial dimensions(H W). The function of this descriptor is to producean embedding of the global distribution of channel-wisefeature responses, allowing information from the globalreceptive field of the network to be used by all its layers.

6 Theaggregation is followed by anexcitationoperation, whichtakes the form of a simple self-gating mechanism that takesthe embedding as input and produces a collection of per-channel modulation weights. These weights are applied tothe feature mapsUto generate the output of the SE blockwhich can be fed directly into subsequent layers of is possible to construct an SE network (SENet) bysimply stacking a collection of SE blocks. Moreover, theseSE blocks can also be used as a drop-in replacement forthe original block at a range of depths in the networkarchitecture (Sec. ). While the template for the buildingblock is generic, the role it performs at different depthsdiffers throughout the network .

7 In earlier layers, it excitesinformative features in a class-agnostic manner, strengthen-ing the shared low-level representations. In later layers, theSE blocks become increasingly specialised, and respond todifferent inputs in a highlyclass-specificmanner (Sec. ). [ ] 25 Oct 20182 Fig. 1. A Squeeze-and-Excitation a consequence, the benefits of the feature recalibrationperformed by SE blocks can be accumulated through design and development of new CNN architecturesis a difficult engineering task, typically requiring the se-lection of many new hyperparameters and layer configura-tions. By contrast, the structure of the SE block is simple andcan be used directly in existing state-of-the-art architecturesby replacing components with their SE counterparts, wherethe performance can be effectively enhanced.

8 SE blocks arealso computationally lightweight and impose only a slightincrease in model complexity and computational provide evidence for these claims, in Sec. 4 we de-velop several SENets and conduct an extensive evaluationon the ImageNet 2012 dataset [10]. We also present resultsbeyond ImageNet that indicate that the benefits of ourapproach are not restricted to a specific dataset or making use of SENets, we ranked first in the ILSVRC2017 classification competition. Our best model ensembleachieves error on the test set1. This repre-sents roughly a25%relative improvement when comparedto the winner entry of the previous year (top-5 error ).2 RELATEDWORKD eeper [11] and Inception mod-els [5] showed that increasing the depth of a network couldsignificantly increase the quality of representations thatit was capable of learning.

9 By regulating the distributionof the inputs to each layer, Batch Normalization (BN) [6]added stability to the learning process in deep networksand produced smoother optimisation surfaces [12]. Buildingon these works, ResNets demonstrated that it was pos-sible to learn considerably deeper and stronger networksthrough the use of identity-based skip connections [13], [14].Highway Networks [15] introduced a gating mechanism toregulate the flow of information along shortcut these works, there have been further reformula-tions of the connections between network layers [16], [17],which show promising improvements to the learning andrepresentational properties of deep alternative, but closely related line of research hasfocused on methods to improve the functional form ofthe computational elements contained within a convolutions have proven to be a popular ap-proach for increasing the cardinality of learned transforma-tions [18], [19].

10 More flexible compositions of operators achieved with multi-branch convolutions [5], [6], [20],[21], which can be viewed as a natural extension of thegrouping operator. In prior work, cross-channel correlationsare typically mapped as new combinations of features, ei-ther independently of spatial structure [22], [23] or jointlyby using standard convolutional filters [24] with1 1convolutions. Much of this research has concentrated on theobjective of reducing model and computational complexity,reflecting an assumption that channel relationships can beformulated as a composition of instance-agnostic functionswith local receptive fields. In contrast, we claim that provid-ing the unit with a mechanism to explicitly model dynamic,non-linear dependencies between channels using global in-formation can ease the learning process, and significantlyenhance the representational power of the architecture the worksdescribed above, there is also a rich history of researchthat aims to forgo manual architecture design and insteadseeks to learn the structure of the network of the early work in this domain was conducted intheneuro-evolutioncommunity, which established methodsfor searching across network topologies with evolutionarymethods [25], [26].