Regularization for Deep Learning

1 Regularization for Deep LearningLecture slides for Chapter 7 of Deep Learning Ian Goodfellow 2016-09-27(Goodfellow 2016)Definition Regularization is any modification we make to a Learning algorithm that is intended to reduce its generalization error but not its training error. (Goodfellow 2016)Weight Decay as Constrained OptimizationCHAPTER 7. Regularization FOR DEEP LEARNINGw1w2w wFigure : An illustration of the effect ofL2(or weight decay) Regularization on the valueof the optimalw. The solid ellipses represent contours of equal value of the unregularizedobjective. The dotted circles represent contours of equal value of theL2regularizer. Atthe point w, these competing objectives reach an equilibrium. In the first dimension, theeigenvalue of the Hessian ofJis small. The objective function does not increase muchwhen moving horizontally away fromw .Becausetheobjectivefunctiondoesnotexpre ssastrongpreferencealongthisdirection,th eregularizerhasastrongeffect on this regularizer pullsw1close to zero.

2 In the second dimension, the objective functionis very sensitive to movements away fromw . The corresponding eigenvalue is large,indicating high curvature. As a result, weight decay affects the position directions along which the parameters contribute significantly to reducingthe objective function are preserved relatively intact. In directions that do notcontribute to reducing the objective function, a small eigenvalue of the Hessiantells us that movement in this direction will not significantly increase the of the weight vector corresponding to such unimportant directionsare decayed away through the use of the Regularization throughout far we have discussed weight decay in terms of its effect on the optimizationof an abstract, general, quadratic cost function. How do these effects relate tomachine Learning in particular? We can find out by studying linear regression, amodel for which the true cost function is quadratic and therefore amenable to thesame kind of analysis we have used so far.

3 Applying the analysis again, we willbe able to obtain a special case of the same results, but with the solution nowphrased in terms of the training data. For linear regression, the cost function is233 Figure (Goodfellow 2016)Norm Penalties L1: Encourages sparsity, equivalent to MAP Bayesian estimation with Laplace prior Squared L2: Encourages small weights, equivalent to MAP Bayesian estimation with Gaussian prior(Goodfellow 2016)Dataset AugmentationAffine DistortionNoiseElastic DeformationHorizontal flipRandom TranslationHue Shift(Goodfellow 2016)Multi-Task LearningCHAPTER 7. Regularization FOR DEEP Learning factors. The model can generally be divided into two kinds of parts and parameters (which only benefit from the examples of their taskto achieve good generalization). These are the upper layers of the neuralnetwork in parameters, shared across all the tasks (which benefit from thepooled data of all the tasks).

4 These are the lower layers of the neural networkin (1)h(1)h(2)h(2)h(3)h(3)y(1)y(1)y(2)y(2)h (shared)h(shared)xxFigure : Multi-task Learning can be cast in several ways in deep Learning frameworksand this figure illustrates the common situation where the tasks share a common input butinvolve different target random variables. The lower layers of a deep network (whether itis supervised and feedforward or includes a generative component with downward arrows)can be shared across such tasks, while task-specific parameters (associated respectivelywith the weights into and fromh(1)andh(2))canbelearnedontopofthose yieldingashared representationh(shared).Theunderlyingass umptionisthatthereexistsacommonpool of factors that explain the variations in the inputx, while each task is associatedwith a subset of these factors. In this example, it is additionally assumed that top-levelhidden unitsh(1)andh(2)are specialized to each task (respectively predictingy(1)andy(2)) while some intermediate-level representationh(shared)is shared across all tasks.

5 Inthe unsupervised Learning context, it makes sense for some of the top-level factors to beassociated with none of the output tasks (h(3)): these are the factors that explain some ofthe input variations but are not relevant for predictingy(1)ory(2).Improved generalization and generalization error bounds (Baxter,1995) can beachieved because of the shared parameters, for which statistical strength can be245 Figure (Goodfellow 2016) Learning CurvesCHAPTER 7. Regularization FOR DEEP LEARNING050100150200250 Time (epochs) (negative log-likelihood)Training set lossValidation set lossFigure : Learning curves showing how the negative log-likelihood loss changes overtime (indicated as number of training iterations over the dataset, orepochs). In thisexample, we train a maxout network on MNIST. Observe that the training objectivedecreases consistently over time, but the validation set average loss eventually begins toincrease again, forming an asymmetric U-shaped improved (in proportion with the increased number of examples for theshared parameters, compared to the scenario of single-task models).

6 Of course thiswill happen only if some assumptions about the statistical relationship betweenthe different tasks are valid, meaning that there is something shared across someof the the point of view of deep Learning , the underlying prior belief is thefollowing:among the factors that explain the variations observed in the dataassociated with the different tasks, some are shared across two or more Early StoppingWhen training large models with sufficient representational capacity to overfitthe task, we often observe that training error decreases steadily over time, butvalidation set error begins to rise again. See an example of thisbehavior. This behavior occurs very means we can obtain a model with better validation set error (and thus,hopefully better test set error) by returning to the parameter setting at the point intime with the lowest validation set error.

7 Every time the error on the validation setimproves, we store a copy of the model parameters. When the training algorithmterminates, we return these parameters, rather than the latest parameters. The246 Figure stopping: terminate while validation set performance is better(Goodfellow 2016)Early Stopping and Weight DecayCHAPTER 7. Regularization FOR DEEP LEARNINGw1w2w ww1w2w wFigure : An illustration of the effect of early stopping.(Left)The solid contour linesindicate the contours of the negative log-likelihood. The dashed line indicates the trajectorytaken by SGD beginning from the origin. Rather than stopping at the pointw thatminimizes the cost , early stopping results in the trajectory stopping at an earlier point w.(Right)An illustration of the effect ofL2regularization for comparison. The dashed circlesindicate the contours of theL2penalty, which causes the minimum of the total cost to lienearer the origin than the minimum of the unregularized are going to study the trajectory followed by the parameter vector duringtraining.

8 For simplicity, let us set the initial parameter vector to the origin,3thatisw(0)=0. Let us study the approximate behavior of gradient descent onJbyanalyzing gradient descent on J:w( )=w( 1) rw J(w( 1))( )=w( 1) H(w( 1) w )( )w( ) w =(I H)(w( 1) w ).( )Let us now rewrite this expression in the space of the eigenvectors ofH, exploitingthe eigendecomposition ofH:H=Q Q>, where is a diagonal matrix andQis an orthonormal basis of ( ) w =(I Q Q>)(w( 1) w )( )Q>(w( ) w )=(I )Q>(w( 1) w )( )3 For neural networks, to obtain symmetry breaking between hidden units, we cannot initializeall the parameters to0, However, the argument holds for any otherinitial valuew(0).251 Figure (Goodfellow 2016)Sparse RepresentationsCHAPTER 7. Regularization FOR DEEP LEARNING266664 14119223377775=2666643 12 54 142 3 11 3 15 4 2 3 2312 30 3 54 22 5 1377775266666640200 3037777775y2 RmB2Rm nh2Rn( )In the first expression, we have an example of a sparsely parametrized linearregression model.

9 In the second, we have linear regression with a sparse representa-tionhof the datax. That is,his a function ofxthat, in some sense, representsthe information present inx, but does so with a sparse Regularization is accomplished by the same sorts of mechanismsthat we have used in parameter penalty Regularization of representations is performed by adding to theloss functionJa norm penalty on therepresentation. This penalty is denoted (h). As before, we denote the regularized loss function by J: J( ;X,y)=J( ;X,y)+ (h)( )where 2[0,1)weights the relative contribution of the norm penalty term, withlarger values of corresponding to more as anL1penalty on the parameters induces parameter sparsity, anL1penalty on the elements of the representation induces representational sparsity: (h)=||h||1=Pi|hi|. Of course, theL1penalty is only one choice of penaltythat can result in a sparse representation.]

10 Others include the penalty derived froma Student-tprior on the representation (Olshausen and Field,1996;Bergstra,2011)and KL divergence penalties (Larochelle and Bengio,2008) that are especiallyuseful for representations with elements constrained to lie on the unit al.(2008)andGoodfellowet al.(2009) both provide examples of strategiesbased on regularizing the average activation across several examples,1mPih(i),tobe near some target value, such as a vector with .01 for each approaches obtain representational sparsity with a hard constraint onthe activation values. For example,orthogonal matching pursuit(Patiet al.,1993) encodes an inputxwith the representationhthat solves the constrainedoptimization problemarg minh,khk0<kkx Whk2,( )wherekhk0is the number of non-zero entries ofh. This problem can be solvedefficiently whenWis constrained to be orthogonal. This method is often called255(Goodfellow 2016)BaggingCHAPTER 7.