Priya Goyal Piotr Dollar Ross Girshick Pieter ... - arXiv

1 Accurate, Large Minibatch SGD:Training ImageNet in 1 HourPriya GoyalPiotr Doll arRoss GirshickPieter NoordhuisLukasz WesolowskiAapo KyrolaAndrew TullochYangqing JiaKaiming HeFacebookAbstractDeep learning thrives with large neural networks andlarge , larger networks and largerdatasets result in longer training times that impede re-search and development progress. Distributed synchronousSGD offers a potential solution to this problem by dividingSGD minibatches over a pool of parallel workers. Yet tomake this scheme efficient, the per-worker workload mustbe large, which implies nontrivial growth in the SGD mini-batch size. In this paper, we empirically show that on theImageNet dataset large minibatches cause optimization dif-ficulties, but when these are addressed the trained networksexhibit good generalization.

2 Specifically, we show no lossof accuracy when training with large minibatch sizes up to8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learningrates as a function of minibatch size and develop a newwarmup scheme that overcomes optimization challengesearly in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192on 256 GPUs in one hour, while matching small minibatchaccuracy. Using commodity hardware, our implementationachieves 90% scaling efficiency when moving from 8 to256 GPUs. Our findings enable training visual recognitionmodels on internet- scale data with high IntroductionScale matters.

3 We are in an unprecedented era in AIresearch history in which the increasing data and modelscale is rapidly improving accuracy in computer vision[22, 41, 34, 35, 36, 16], speech [17, 40], and natural lan-guage processing [7, 38]. Take the profound impact in com-puter vision as an example: visual representations learnedby deep convolutional neural networks [23, 22] show excel-lent performance on previously challenging tasks like Ima-geNet classification [33] and can be transferred to difficultperception problems such as object detection and segmen-641282565121k2k4k8k16k32k64kmini- batch size2025303540 ImageNet top-1 validation errorFigure top-1 validation errorvs. minibatch range of plus/minustwostandard deviations is shown.

4 Wepresent a simple and general technique for scaling distributed syn-chronous SGD to minibatches of up to 8k imageswhile maintain-ing the top-1 error of small minibatch training. For all minibatchsizes we set the learning rate as alinearfunction of the minibatchsize and apply a simple warmup phase for the first few epochs oftraining. All other hyper-parameters are kept fixed. Using thissimple approach, accuracy of our models is invariant to minibatchsize (up to an 8k minibatch size). Our techniques enable a lin-ear reduction in training time with 90% efficiency as we scaleto large minibatch sizes, allowing us to train an accurate 8k mini-batch ResNet-50 model in 1 hour on 256 [8, 10, 28]. Moreover, this pattern generalizes: largerdatasets and neural network architectures consistently yieldimproved accuracy across all tasks that benefit from pre-training [22, 41, 34, 35, 36, 16].

5 But as model and datascale grow, so does training time; discovering the potentialand limits of large- scale deep learning requires developingnovel techniques to keep training time goal of this report is to demonstrate the feasibility of,and to communicate a practical guide to, large- scale train-ing with distributedsynchronousstochastic gradient descent(SGD). As an example, we scale ResNet-50 [16] training,originally performed with a minibatch size of 256 images(using 8 Tesla P100 GPUs, training time is 29 hours), tolarger minibatches (see Figure 1). In particular, we showthatwith a large minibatch size of 8192, we can trainResNet-50 in 1 hour using 256 GPUs while maintaining1 [ ] 30 Apr 2018the same level of accuracy as the 256 minibatch distributed synchronous SGD is now commonplace,no existing results show that generalization accuracy can bemaintained with minibatches as large as 8192 or that suchhigh-accuracy models can be trained in such short tackle this unusually large minibatch size, we employa simple and hyper-parameter-freelinear scaling ruleto ad-just the learning rate.

6 While this guideline is found in ear-lier work [21, 4], its empirical limits are not well under-stood and informally we have found that it is not widelyknown to the research community. To successfully applythis rule, we present a newwarmupstrategy, , a strategyof using lower learning rates at the start of training [16], toovercome early optimization difficulties. Importantly, notonly does our approach match the baselinevalidationerror,but also yields training error curves that closely match thesmall minibatch are presented in comprehensive experiments in 5 show thatopti-mizationdifficulty is the main issue with large minibatches,rather than poorgeneralization(at least on ImageNet), incontrast to some recent studies [20].

7 Additionally, we showthat the linear scaling rule and warmup generalize to morecomplex tasks including object detection and instance seg-mentation [9, 31, 14, 28], which we demonstrate via therecently developed Mask R-CNN [14]. We note that a ro-bust and successful guideline for addressing a wide rangeof minibatch sizes has not been presented in previous the strategy we deliver is simple, its successfulapplication requires correct implementation with respect toseemingly minor and often not well understood implemen-tation details within deep learning libraries. Subtleties in theimplementation of SGD can lead to incorrect solutions thatare difficult to discover. To provide more helpful guidancewe describe common pitfalls and the relevant implementa-tion details that can trigger these traps in strategy applies regardless of framework, butachieving efficient linear scaling requires nontrivial com-munication algorithms.

8 We use the open-sourceCaffe21deep learning framework andBig BasinGPU servers [24],which operates efficiently using standard Ethernet network-ing (as opposed to specialized network interfaces). We de-scribe the systems algorithms that enable our approach tooperate near its full potential in practical advances described in this report are help-ful across a range of domains. In an industrial domain,our system unleashes the potential of training visual modelsfrom internet- scale data, enabling training withbillions ofimages per day. Of equal importance, in a research domain,we have found it to simplify migrating algorithms from asingle-GPU to a multi-GPU implementation without requir-ing hyper-parameter search, in our experience migrat-ing Faster R-CNN [31] and ResNets [16] from 1 to 8 Large Minibatch SGDWe start by reviewing the formulation of Stochastic Gra-dient Descent (SGD), which will be the foundation of ourdiscussions in the following sections.

9 We consider super-vised learning by minimizing a lossL(w)of the form:L(w) =1|X| x Xl(x,w).(1)Hereware the weights of a network,Xis a labeled trainingset, andl(x,w)is the loss computed from samplesx Xand their labelsy. Typicallylis the sum of a classificationloss ( , cross-entropy) and a regularization loss Stochastic Gradient Descent[32], usually re-ferred to as simply as SGD in recent literature even thoughit operates on minibatches, performs the following update:wt+1=wt 1n x B l(x,wt).(2)HereBis aminibatchsampled fromXandn=|B|is theminibatch size, is thelearning rate, andtis the iterationindex. Note that in practice we use momentum SGD; wereturn to a discussion of momentum in Learning Rates for Large MinibatchesOur goal is to use large minibatches in place of smallminibatches whilemaintaining training and generalizationaccuracy.

10 This is of particular interest in distributed learn-ing, because it can allow us to scale to multiple workers2us-ing simple data parallelism without reducing the per-workerworkload and without sacrificing model we will show in comprehensive experiments, wefound that the following learning rate scaling rule is sur-prisingly effective for a broad range of minibatch sizes:Linear Scaling Rule:When the minibatch size ismultiplied byk, multiply the learning rate other hyper-parameters (weight decay,etc.) are kept un-changed. As we will show in 5, thelinear scaling rulecanhelp us to not only match the accuracy between using smalland large minibatches, but equally importantly, to largelymatch their training curves, which enables rapid debuggingand comparison of experiments prior to present an informal discussion of thelinear scaling rule and why it may be effective.