Lecture notes on ridge regression - arXiv

1 Lecture notes on ridge regression Version , January 18, 2020. [ ] 18 Jan 2020. Wessel N. van Wieringen1,2. 1. Department of Epidemiology and Biostatistics, Amsterdam Public Health research institute, Amsterdam UMC, location VUmc Box 7057, 1007 MB Amsterdam, The Netherlands 2. Department of Mathematics, VU University Amsterdam De Boelelaan 1111, 1081 HV Amsterdam, The Netherlands Email: License This document is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike license: Disclaimer This document is a collection of many well-known results on ridge regression . The current status of the document is work-in-progress' as it is incomplete (more results from literature will be included) and it may contain incon- sistencies and errors.

2 Hence, reading and believing at own risk. Finally, proper reference to the original source may sometimes be lacking. This is regrettable and these references if ever known to the author will be included in later versions. Acknowledgements Many people aided in various ways to the construction of these notes . Mark A. van de Wiel commented on various parts of Chapter 2. Jelle J. Goeman clarified some matters behind the method described in Section Paul Eilers pointed to helpful references for Chapter 4 and provided parts of the code used in Section Small typo's or minor errors, that have hopefully been corrected in the latest version, were pointed out by (among others): Rikkert Hindriks, Micah Blake McCurdy, Jose P. Gonza lez-Brenes, and numerous students from the High-dimensional data analysis- and Statistics for high-dimensional data-courses taught at Leiden University and the VU University Amsterdam, respectively.

3 Contents 1 ridge regression 2. Linear regression 2. The ridge regression estimator 5. Eigenvalue shrinkage 9. Principal components regression 10. Moments 10. Expectation 10. Variance 11. Mean squared error 14. Constrained estimation 16. Degrees of freedom 19. Efficient calculation 19. Choice of the penalty parameter 20. Information criterion 20. Cross-validation 21. Generalized cross-validation 22. Simulations 23. Role of the variance of the covariates 23. ridge regression and collinearity 25. Variance inflation factor 27. Illustration 29. MCM7 expression regulation by microRNAs 29. Conclusion 33. Exercises 34. 2 Bayesian regression 36. A minimum of prior knowledge on Bayesian statistics 36. Relation to ridge regression 37. Markov chain Monte Carlo 40.

4 Empirical Bayes 45. Conclusion 46. Exercises 46. 3 Generalizing ridge regression 47. Moments 48. The Bayesian connection 49. Application 50. Generalized ridge regression 52. Conclusion 53. Exercises 53. 4 Mixed model 55. Link to ridge regression 60. REML consistency, high-dimensionally 61. Illustration: P-splines 63. 5 ridge logistic regression 67. Logistic regression 67. ridge estimation 69. Moments 71. The Bayesian connection 73. Penalty parameter selection 74. Application 74. Conclusion 76. Exercises 76. 6 Lasso regression 79. Uniqueness 80. Analytic solutions 82. Sparsity 85. Maximum number of selected covariates 87. Estimation 88. Quadratic programming 88. Iterative ridge 89. Gradient ascent 90. Coordinate descent 92. Moments 92. The Bayesian connection 93.

5 Comparison to ridge 95. Linearity 95. Shrinkage 95. Simulation I: Covariate selection 96. Simulation II: correlated covariates 97. Pandora's box 97. Elastic net 97. Fused lasso 99. The (sparse) group lasso 101. Adaptive lasso 102. The 0 penalty 103. Exercises 103. 1 ridge regression High-throughput techniques measure many characteristics of a single sample simultaneously. The number of characteristics p measured may easily exceed ten thousand. In most medical studies the number of samples n involved often falls behind the number of characteristics measured, : p > n. The resulting (n p)-dimensional data matrix X: . X1, X1,1 .. X1,p .. X = (X ,1 | .. | X ,p ) = .. = .. Xn, Xn,1 .. Xn,p from such a study contains a larger number of covariates than samples.

6 When p > n the data matrix X is said to be high-dimensional. In this chapter we adopt the traditional statistical notation of the data matrix. An alternative notation would be X (rather than X), which is employed in the field of (statistical) bioinformatics. In X the rows comprise the samples rather than the covariates. The case for the bioinformatics notation stems from practical arguments. A. spreadsheet is designed to have more rows than columns. In case p > n the traditional notation yields a spreadsheet with more columns than rows. When p > 10000 the conventional display is impractical. In this chapter we stick to the conventional statistical notation of the data matrix as all mathematical expressions involving X are then in line with those of standard textbooks on regression .

7 The information contained in X is often used to explain a particular property of the samples involved. In applications in molecular biology X may contain microRNA expression data from which the expression levels of a gene are to be described. When the gene's expression levels are denoted by Y = (Y1 , .. , Yn ) , the aim is to find the linear relation Yi = Xi, from the data at hand by means of regression analysis. regression is however frustrated by the high-dimensionality of X (illustrated in Section and at the end of Section ). These notes discuss how regression may be modified to accommodate the high-dimensionality of X. First, however, standard'. linear regression is recaputilated. Linear regression Consider an experiment in which p characteristics of n samples are measured.

8 The data from this experiment are denoted X, with X as above. The matrix X is called the design matrix. Additional information of the samples is available in the form of Y (also as above). The variable Y is generally referred to as the response variable. The aim of regression analysis is to explain Y in terms of X through a functional relationship like Yi = f (Xi, ). When no prior knowledge on the form of f ( ) is available, it is common to assume a linear relationship between X and Y. This assumption gives rise to the linear regression model: Yi = Xi, + i ( ). = 1 Xi,1 + .. + p Xi,p + i . In model ( ) = ( 1 , .. , p ) is the regression parameter. The parameter j , j = 1, .. , p, represents the effect size of covariate j on the response. That is, for each unit change in covariate j (while keeping the other covariates fixed) the observed change in the response is equal to j.

9 The second summand on the right- hand side of the model, i , is referred to as the error. It represents the part of the response not explained by the functional part Xi, of the model ( ). In contrast to the functional part, which is considered to be systematic ( non-random), the error is assumed to be random. Consequently, Yi1 , need not equal Yi2 , for i1 6= i2 , even if Xi1 , = Xi2 , . To complete the formulation of model ( ) we need to specify the probability distribution of Linear regression 3. i . It is assumed that i N (0, 2 ) and the i are independent, : . 2 if i1 = i2 , Cov( i1 , i2 ) =. 0 if i1 6= i2 . The randomness of i implies that Yi is also a random variable. In particular, Yi is normally distributed, because i N (0, 2 ) and Xi, is a non-random scalar.

10 To specify the parameters of the distribution of Yi we need to calculate its first two moments. Its expectation equals: E(Yi ) = E(Xi, ) + E( i ) = Xi, , while its variance is: Var(Yi ) = E{[Yi E(Yi )]2 } = E(Yi2 ) [E(Yi )]2. = E[(Xi, )2 + 2 i Xi, + 2i ] (Xi, )2. = E( 2i ) = Var( i ) = 2 . Hence, Yi N (Xi, , 2 ). This formulation (in terms of the normal distribution) is equivalent to the formulation of model ( ), as both capture the assumptions involved: the linearity of the functional part and the normality of the error. Model ( ) is often written in a more condensed matrix form: Y = X + , ( ). where = ( 1 , 2 , .. , n ) and distributed as N (0p , 2 Inn ). As above model ( ) can be expressed as a multivariate normal distribution: Y N (X , 2 Inn ).