Improved Adam Optimizer for Deep Neural Networks

1 Improved Adam Optimizer forDeep Neural NetworksZijun ZhangDepartment of Computer ScienceUniversity of Adaptive optimization algorithms, such as Adamand RMSprop, have witnessed better optimization performancethan stochastic gradient descent (SGD) in some , recent studies show that they often lead to worsegeneralization performance than SGD, especially for trainingdeep Neural Networks (DNNs). In this work, we identify thereasons that Adam generalizes worse than SGD, and developa variant of Adam to eliminate the generalization gap. Theproposed method, normalized direction-preserving Adam (ND-Adam), enables more precise control of the direction and stepsize for updating weight vectors, leading to significantly improvedgeneralization performance. Following a similar rationale, wefurther improve the generalization performance in classificationtasks by regularizing the softmax logits.

2 By bridging the gapbetween SGD and Adam, we also hope to shed light on whycertain optimization algorithms generalize better than INTRODUCTIONIn contrast with the growing complexity of Neural networkarchitectures [1, 2, 3], the training methods remain relativelysimple. Most practical optimization methods for deep neuralnetworks (DNNs) are based on the stochastic gradient descent(SGD) algorithm. However, the learning rate of SGD, as ahyperparameter, is often difficult to tune, since the magnitudesof different parameters vary widely, and adjustment is requiredthroughout the training tackle this problem, several adaptive variants of SGDwere developed, including Adagrad [4], Adadelta [5], RM-Sprop [6], Adam [7]. These algorithms aim to adapt thelearning rate to different parameters automatically, based onthe statistics of gradient.

3 Although they usually simplifylearning rate settings, and lead to faster convergence, it isobserved that their generalization performance tend to besignificantly worse than that of SGD in some scenarios [8].This intriguing phenomenon may explain why SGD (possiblywith momentum) is still prevalent in training state-of-the-art deep models, especially feedforward DNNs [1, 2, 3].Furthermore, recent work has shown that DNNs are capableof fitting noise data [9], suggesting that their generalizationcapabilities are not the mere result of DNNs themselves, butare entwined with optimization [10].This work aims to bridge the gap between SGD and Adamin terms of the generalization performance. To this end, weidentify two problems that may degrade the generalizationperformance of Adam, and show how these problems are(partially) avoided by using SGD with L2 weight decay.

4 First,the updates of SGD lie in the span of historical gradients,whereas it is not the case for Adam. This difference has beendiscussed in rather recent literature [8], where the authors showthat adaptive methods can find drastically different but worsesolutions than SGD. Second, while the magnitudes of Adamparameter updates are invariant to rescaling of the gradient,the effect of the updates on thesameoverall network functionstill varies with the magnitudes of parameters. As a result,the effective learning rates of weight vectors tend to decreaseduring training, which leads to sharp local minima that do notgeneralize well [11].To address these two problems of Adam, we propose thenormalized direction-preserving Adam (ND-Adam) algorithm,which controls the update direction and step size in a moreprecise way. We show that ND-Adam is able to achievesignificantly better generalization performance than vanillaAdam, and matches that of SGD in image classification summarize our contributions as follows: We observe that the directions of Adam parameter up-dates are different from that of SGD, , Adam does notpreserve the directions of gradients as SGD does.

5 We fixthe problem by adapting the learning rate to each weightvector, instead of each individual weight, such that thedirection of the gradient is preserved. For both Adam and SGD without L2 weight decay, weobserve that the magnitude of each vector s directionchange depends on its L2-norm. We show that, usingSGD with L2 weight decay implicitly normalizes theweight vectors, and thus remove the dependence in anapproximate manner. We fix the problem for Adam byexplicitly normalizing each weight vector, and by opti-mizing only its direction, such that the effective learningrate can be precisely controlled. We further demonstrate that, without proper regulariza-tion, the learning signal backpropagated from the softmaxlayer may vary with the overall magnitude of the logits inan undesirable way. Based on the observation, we applybatch normalization or L2-regularization to the logits,which further improves the generalization performancein classification NORMALIZEDDIRECTION-PRESERVINGADAMWe present the normalized direction-preserving Adam (ND-Adam) algorithm, which essentially improves the optimization978-1-5386-2542-2/18/$ 2018 IEEEof the input weights of hidden units, while employing thevanilla Adam algorithm to update other parameters.

6 Specif-ically, we divide the trainable parameters, , into two sets, vand s, such that v={wi|i N }, and s={ \ v}. Thenwe update vand sby different rules, as described by Alg. learning rates for the two sets of parameters are denotedby vtand st, 1:Normalized direction-preserving Adam/*Initialization*/t 0;fori Ndowi,0 wi,0/ wi,0 2;m0(wi) 0;v0(wi) 0;/*PerformTiterations of training*/whilet < Tdot t+ 1;/*Update v*/fori Ndo gt(wi) L/ wi;gt(wi) gt(wi) ( gt(wi) wi,t 1)wi,t 1;mt(wi) 1mt 1(wi) + (1 1)gt(wi);vt(wi) 2vt 1(wi) + (1 2) gt(wi) 22; mt(wi) mt(wi)/(1 t1); vt(wi) vt(wi)/(1 t2); wi,t wi,t 1 vt mt(wi)/( vt(wi) + );wi,t wi,t/ wi,t 2;/*Update susing Adam*/ st AdamUpdate ( st 1; st, 1, 2);return T;As shown in Alg. 1, the proposed algorithm adapts thelearning rate to each input weight vector, instead of eachindividual weight, such that the direction of the gradient ispreserved.

7 In addition, it strictly keeps the magnitude of eachweight vector constant, and optimizes only its direction, suchthat the effective learning rate can be precisely evaluate the performance of ND-Adam, we use SGD andND-Adam to train wide residual Networks (WRN) [12] on theCIFAR-10 and CIFAR-100 datasets, and summarize the resultsin Table ITEST ERROR RATES (%)CIFAR-100 Error (%) CONCLUSIONWe introduced ND-Adam, a tailored version of Adam fortraining DNNs, to bridge the generalization gap between Adamand SGD. ND-Adam is designed to preserve the direction ofgradient for each weight vector, and produce the regularizationeffect of L2 weight decay in a more precise and principledway. We further introduced regularized softmax, which limitsthe magnitude of softmax logits to provide better learningsignals. Combining ND-Adam and regularized softmax, weshow through experiments significantly Improved generaliza-tion performance, eliminating the gap between Adam [1] C.

8 Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabi-novich, Going deeper with convolutions, inProceed-ings of the IEEE conference on computer vision andpattern recognition, 2015, pp. 1 9.[2] K. He, X. Zhang, S. Ren, and J. Sun, Deep residuallearning for image recognition, inProceedings of theIEEE conference on computer vision and pattern recog-nition, 2016, pp. 770 778.[3] J. Hu, L. Shen, and G. Sun, squeeze -and-excitationnetworks, arXiv preprint , 2017.[4] J. Duchi, E. Hazan, and Y. Singer, Adaptive subgradientmethods for online learning and stochastic optimization, Journal of Machine Learning Research, vol. 12, no. Jul,pp. 2121 2159, 2011.[5] M. D. Zeiler, Adadelta: an adaptive learning ratemethod, arXiv preprint , 2012.[6] T. Tieleman and G.

9 Hinton, Lecture RmsProp:Divide the gradient by a running average of its recentmagnitude, COURSERA: Neural Networks for MachineLearning, 2012.[7] D. Kingma and J. Ba, Adam: A method for stochasticoptimization, arXiv preprint , 2014.[8] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, andB. Recht, The marginal value of adaptive gradientmethods in machine learning, inAdvances in NeuralInformation Processing Systems, 2017.[9] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, Understanding deep learning requires rethinking gener-alization, inICLR 2017, 2017.[10] D. Arpit, S. Jastrz ebski, N. Ballas, D. Krueger, E. Bengio,M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville,Y. Bengioet al., A closer look at memorization in deepnetworks, arXiv preprint , 2017.[11] S. Hochreiter and J. Schmidhuber, Flat minima, NeuralComputation, vol.

10 9, no. 1, pp. 1 42, 1997.[12] S. Zagoruyko and N. Komodakis, Wide residual net-works, arXiv preprint , 2016.