RepVGG: Making VGG-Style ConvNets Great Again

1 RepVGG: Making VGG-Style ConvNets Great AgainXiaohan Ding1 Xiangyu Zhang2 Ningning Ma3 Jungong Han4 Guiguang Ding1 Jian Sun21 Beijing National Research Center for Information Science and Technology (BNRist);School of Software, Tsinghua University, Beijing, China2 MEGVII Technology3 Hong Kong University of Science and Technology4 Computer Science Department, Aberystwyth University, SY23 3FL, present a simple but powerful architecture of convo-lutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of3 3con-volution and ReLU, while the training-time model has amulti-branch topology. Such decoupling of the training-time and inference-time architecture is realized by a struc-tural re-parameterization technique so that the model isnamed RepVGG. On ImageNet, RepVGG reaches over 80%top-1 accuracy, which is the first time for a plain model,to the best of our knowledge. On NVIDIA 1080Ti GPU,RepVGG models run 83% faster than ResNet-50 or 101%faster than ResNet-101 with higher accuracy and show fa-vorable accuracy-speed trade-off compared to the state-of-the-art models like EfficientNet and RegNet.

2 The codeand trained models are available IntroductionA classic Convolutional Neural Network (ConvNet),VGG [31], achieved huge success in image recognition witha simple architecture composed of a stack of conv, ReLU,and pooling. With Inception [33,34,32,19], ResNet [12]and DenseNet [17], a lot of research interests were shiftedto well-designed architectures, Making the models more andmore complicated. Some recent architectures are based on This work is supported by The National Key Research and Develop-ment Program of China (No. 2017 YFA0700800), the National NaturalScience Foundation of China ( , ) and BeijingAcademy of Artificial Intelligence (BAAI). Xiaohan Ding is funded by theBaidu Scholarship Program 2019. This work is done during Xiaohan Dingand Ningning Ma s internship at MEGVII Technology. Corresponding (examples/second)747576777879accuracyRep VGGRegNetXResNetResNeXtEfficientNet20030 0400500600speed (examples/second) 1: Top-1 accuracy on ImageNetvs.

3 Actual : lightweight and middleweight RepVGG and baselinestrained in 120 epochs. Right: heavyweight models trainedin 200 epochs. The speed is tested on the same 1080Ti witha batch size of 128, full precision (fp32), single crop, andmeasured in examples/second. The input resolution is 300for EfficientNet-B3 [35] and 224 for the [44,29,23] or manual [28] architecture search,or a searched compound scaling strategy [35].Though many complicated ConvNets deliver higher ac-curacy than the simple ones, the drawbacks are )The complicated multi-branch designs ( , residual-addition in ResNet and branch-concatenation in Inception)make the model difficult to implement and customize, slowdown the inference and reduce the memory )Some components ( , depthwise conv in Xception [3]and MobileNets [16,30] and channel shuffle in ShuffleNets[24,41]) increase the memory access cost and lack sup-ports of various devices. With so many factors affectingthe inference speed, the amount of floating-point opera-tions (FLOPs) does not precisely reflect the actual some novel models have lower FLOPs than the old-fashioned ones like VGG and ResNet-18/34/50 [12], they13733+ReLU(B) RepVGG trainingIdentity3 31 1+3 31 1+3 31 1+3 31 1conv(C) RepVGG inference3 33 33 33 33 3+3 33 3+3 31 1(A) ResNetstride=2stride=2stride=2 Figure 2: Sketch of RepVGG architecture.

4 RepVGG has5 stages and conducts down-sampling via stride-2 convolu-tion at the beginning of a stage. Here we only show the first4 layers of a specific stage. As inspired by ResNet [12], wealso use identity and1 1branches, but only for not run faster ( ). Consequently, VGG and theoriginal versions of ResNets are still heavily used for real-world applications in both academia and this paper , we propose RepVGG, a VGG-Style archi-tecture which outperforms many complicated models ( ). RepVGG has the following advantages. The model has a VGG-like plain ( feed-forward)topology1without any branches, which means everylayer takes the output of its only preceding layer asinput and feeds the output into its only following layer. The model s body uses only3 3conv and ReLU. The concrete architecture (including the specific depthand layer widths) is instantiated with no automaticsearch [44], manual refinement [28], compound scal-ing [35], nor other heavy is challenging for a plain model to reach a comparablelevel of performance as the multi-branch architectures.

5 Anexplanation is that a multi-branch topology, , ResNet,makes the model an implicit ensemble of numerous shal-lower models [36], so that training a multi-branch modelavoids the gradient vanishing the benefits of multi-branch architecture are allfor training and the drawbacks are undesired for infer-ence, we propose todecouple the training-time multi-branch and inference-time plain architectureviastructuralre-parameteriza tion, which means converting the architec-ture from one to another via transforming its be specific, a network structure is coupled with a set ofparameters, , a conv layer is represented by a 4th-order1In this paper , a networktopologyonly focuses on how the componentsconnect to others, anarchitecturerefers to the topology together with thespecification of components like depth and width, and astructuremay referto any component or part of the tensor. If the parameters of a certain structure can beconverted into another set of parameters coupled by anotherstructure, we can equivalently replace the former with thelatter, so that the overall network architecture is , we construct the training-time RepVGG us-ing identity and1 1branches, which is inspired by ResNetbut in a different way that the branches can be removed bystructural re-parameterization ( ,4).

6 After training, weperform the transformation with simple algebra, as an iden-tity branch can be regarded as a degraded1 1conv, and thelatter can be further regarded as a degraded3 3conv, sothat we can construct a single3 3kernel with the trainedparameters of the original3 3kernel, identity and1 1branches and batch normalization (BN) [19] layers. Conse-quently, the transformed model has a stack of3 3convlayers, which is saved for test and , the body of an inference-time RepVGG onlyhas one single type of operator:3 3conv followed byReLU, which makes RepVGG fast on generic computingdevices like GPUs. Even better, RepVGG allows for spe-cialized hardware to achieve even higher speed becausegiven the chip size and power consumption, the fewer typesof operators we require, the more computing units we canintegrate onto the chip. Consequently, an inference chipspecialized for RepVGG can have an enormous number of3 3-ReLU units and fewer memory units (because the plaintopology is memory-economical, as shown in ).

7 Ourcontributions are summarized as follows. We propose RepVGG, a simple architecture withfavorable speed-accuracy trade-off compared to thestate-of-the-arts. We propose to use structural re-parameterization to de-couple a training-time multi-branch topology with aninference-time plain architecture. We show the effectiveness of RepVGG in image classi-fication and semantic segmentation, and the efficiencyand ease of Related From Single-path to Multi-branchAfter VGG [31] raised the top-1 accuracy of ImageNetclassification to above 70%, there have been many inno-vations in Making ConvNets complicated for high perfor-mance, , the contemporary GoogLeNet [33] and laterInception models [34,32,19] adopted elaborately designedmulti-branch architectures, ResNet [12] proposed a sim-plified two-branch architecture, and DenseNet [17] madethe topology more complicated by connecting lower-levellayers with numerous higher-level ones. Neural architec-ture search (NAS) [44,29,23,35] and manual designingspace design [28] can generate ConvNets with higher per-formance but at the costs of vast computing resources or13734manpower.

8 Some large versions of NAS-generated modelsare even not trainable on ordinary GPUs, hence limiting theapplications. Except for the inconvenience of implemen-tation, the complicated models may reduce the degree ofparallelism [24] hence slow down the Effective Training of Single-path ModelsThere have been some attempts to train ConvNets with-out branches. However, the prior works mainly sought tomake the very deep models converge with reasonable ac-curacy, but not achieve better performance than the com-plicated models. Consequently, the methods and resul-tant models were neither simple nor practical. An initial-ization method [37] was proposed to train extremely deepplain ConvNets . With a mean-field-theory-based scheme,10,000-layer networks were trained over 99% accuracy onMNIST and 82% on CIFAR-10. Though the models werenot practical (even LeNet-5 [21] can reach accuracyon MNIST and VGG-16 can reach above 93% on CIFAR-10), the theoretical contributions were insightful.

9 A recentwork [25] combined several techniques including LeakyReLU, max-norm and careful initialization. On ImageNet,it showed that a plain ConvNet with 147M parameters couldreach top-1 accuracy, which was 2% lower than itsreported baseline (ResNet-101, , 45M parameters).Notably, this paper is not merely a demonstration thatplain models can converge reasonably well, and does notintend to train extremely deep ConvNets like , we aim to build a simple model with reasonabledepth and favorable accuracy-speed trade-off, which canbe simply implemented with the most common components( , regular conv and BN) and simple Model Re-parameterizationDiracNet [39] is a re-parameterization method relatedto ours. It builds deep plain models by encoding the ker-nel of a conv layer as W =diag(a)I +diag(b)Wnorm,where Wis the eventual weight used for convolution (a 4th-order tensor viewed as a matrix),aandbare learned vec-tors, andWnormis the normalized learnable kernel.

10 Com-pared to ResNets with comparable amount of parameters,the top-1 accuracy of DiracNet is lower on CIFAR-100 ( ) and lower on ImageNet( of DiracNet-34vs. of ResNet-34). Dirac-Net differs from our method in two )The training-time behavior of RepVGG is implemented by the actualdataflow through a concrete structure which can be laterconverted into another, while DiracNet merely uses an-other mathematical expression of conv kernels for easieroptimization. In other words, a training-time RepVGGis a real multi-branch model, but a DiracNet is )The performance of a DiracNet is higher than a normallyparameterized plain model but lower than a comparableResNet, while RepVGG models outperform ResNets by alarge margin. Asym Conv Block (ACB) [10], DO-Conv [1]and ExpandNet [11] can also be viewed as structural re-parameterization in the sense that they convert a block intoa conv. Compared to our method, the difference is that theyare designed for component-level improvements and usedas a drop-in replacement for conv layers in any architec-ture, while our structural re-parameterization is critical fortraining plain ConvNets , as shown in Winograd ConvolutionRepVGG uses only3 3conv because it is highly op-timized by some modern computing libraries like NVIDIAcuDNN [2] and Intel MKL [18] on GPU and CPU.