for Chapter Generalized Linear Models (GLMs) - MIT …

1 Statistics for Applications Chapter 10: Generalized Linear Models (GLMs). 1/52. Linear model A Linear model assumes Y |X N ( (X), 2 I), And IE(Y |X) = (X) = X , 2/52. Components of a Linear model The two components (that we are going to relax) are 1. Random component: the response variable Y |X is continuous and normally distributed with mean = (X) = IE(Y |X). 2. Link: between the random and covariates X = (X (1) , X (2) , , X (p) ) : (X) = X . 3/52. Generalization A Generalized Linear model (GLM) generalizes normal Linear regression Models in the following directions. 1. Random component: Y some exponential family distribution 2.

2 Link: between the random and covariates: g (X) = X .. where g called link function and = IE(Y |X). 4/52. Example 1: Disease Occuring Rate In the early stages of a disease epidemic, the rate at which new cases occur can often increase exponentially through time. Hence, if i is the expected number of new cases on day ti , a model of the form i = exp( ti ). seems appropriate. Such a model can be turned into GLM form, by using a log link so that log( i ) = log( ) + ti = 0 + 1 ti . Since this is a count, the Poisson distribution (with expected value i ) is probably a reasonable distribution to try. 5/52. Example 2: Prey Capture Rate(1).

3 The rate of capture of preys, yi , by a hunting animal, tends to increase with increasing density of prey, xi , but to eventually level o , when the predator is catching as much as it can cope with. A suitable model for this situation might be xi i = , h + xi where represents the maximum capture rate, and h represents the prey density at which the capture rate is half the maximum rate. 6/52. Example 2: Prey Capture Rate (2). 7/52. Example 2: Prey Capture Rate (3). Obviously this model is non- Linear in its parameters, but, by using a reciprocal link, the right-hand side can be made Linear in the parameters, 1 1 h 1 1. g( i ) = = + = 0 + 1.

4 I xi xi The standard deviation of capture rate might be approximately proportional to the mean rate, suggesting the use of a Gamma distribution for the response. 8/52. Example 3: Kyphosis Data The Kyphosis data consist of measurements on 81 children following corrective spinal surgery. The binary response variable, Kyphosis, indicates the presence or absence of a postoperative deforming. The three covariates are, Age of the child in month, Number of the vertebrae involved in the operation, and the Start of the range of the vertebrae involved. The response variable is binary so there is no choice: Y |X is Bernoulli with expected value (X) (0, 1).

5 We cannot write (X) = X . because the right-hand side ranges through IR. We need an invertible function f such that f (X ) (0, 1). 9/52. GLM: motivation clearly, normal LM is not appropriate for these examples;. need a more general regression framework to account for various types of response data Exponential family distributions develop methods for model tting and inferences in this framework Maximum Likelihood estimation. 10/52. Exponential Family A family of distribution {P : }, IRk is said to be a k-parameter exponential family on IRq , if there exist real valued functions: 1 , 2 , , k and B of , T1 , T2 , , Tk , and h of x IRq such that the density function (pmf or pdf) of P can be written as Lk p (x) = exp[ i ( )Ti (x) B( )]h(x).

6 I=1. 11/52. Normal distribution example Consider X N ( , 2 ), = ( , 2 ). The density is ( 1 2 2 ) 1. p (x) = exp x x , 2 2 2 2 2 2 . which forms a two-parameter exponential family with 1. 1 = 2. , 2 = 2 , T1 (x) = x, T2 (x) = x2 , 2 . 2 . B( ) = 2. + log( 2 ), h(x) = 1. 2 . When 2 is known, it becomes a one-parameter exponential family on IR: x2. 2 e 2 2. = 2 , T (x) = x, B( ) = 2 , h(x) = . 2 2 . 12/52. Examples of discrete distributions The following distributions form discrete exponential families of distributions with pmf Bernoulli(p): px (1 p)1 x , x {0, 1}. x . Poisson( ): e , x = 0, 1, .. x! 13/52. Examples of Continuous distributions The following distributions form continuous exponential families of distributions with pdf: 1 x Gamma(a, b): a xa 1 e b.

7 (a)b above: a: shape parameter, b: scale parameter reparametrize: = ab: mean parameter ( )a 1 a xa 1 e . ax (a) . 1 /x Inverse Gamma( , ): x e . ( ). 2 2. 2 2 (x ). Inverse Gaussian( , 2 ): e 2x . 2 x3. Others: Chi-square, Beta, Binomial, Negative binomial distributions. 14/52. Components of GLM. 1. Random component: Y some exponential family distribution 2. Link: between the random and covariates: g (X) = X .. where g called link function and (X) = IE(Y |X). 15/52. One-parameter canonical exponential family Canonical exponential family for k = 1, y IR. ( y b( ) ). f (y) = exp + c(y, ).. for some known functions b( ) and c( , ).

8 If is known, this is a one-parameter exponential family with being the canonical parameter . If is unknown, this may/may not be a two-parameter exponential family. is called dispersion parameter. In this class, we always assume that is known. 16/52. Normal distribution example Consider the following Normal density function with known variance 2 , 1 (y )2. f (y) = e 2 2. 2 . y 12 2 1 y2. ( ) . 2. = exp + log(2 ) , 2 2 2. 2. Therefore = , = 2 , , b( ) = 2 , and 1 y2. c(y, ) = ( + log(2 )). 2 . 17/52. Other distributions Table 1: Exponential Family Normal Poisson Bernoulli Notation N ( , 2 ) P( ) B(p). Range of y ( , ) [0, ) {0, 1}.]

9 2 1 1. 2. b( ) 2 e log(1 + e ). 2. c(y, ) 21 ( y + log(2 )) log y! 1. 18/52. Likelihood Let ( ) = log f (Y ) denote the log-likelihood function. The mean IE(Y ) and the variance var(Y ) can be derived from the following identities First identity . IE( ) = 0, . Second identity 2 . IE( 2. ) + IE( )2 = 0.. Obtained from f (y)dy 1 . 19/52. Expected value Note that Y b( . ( ) = + c(Y ; ), . Therefore Y b ( ). =.. It yields IE(Y ) b ( )). 0 = IE( )= , . which leads to IE(Y ) = = b ( ). 20/52. Variance On the other hand we have we have 2 2 b ( ) ( Y b ( ) )2. + ( ) = +. 2 . and from the previous result, Y b ( ) Y IE(Y ). =.. Together, with the second identity, this yields b ( ) var(Y ).

10 0= + , 2. which leads to var(Y ) = V (Y ) = b ( ) . 21/52. Example: Poisson distribution Example: Consider a Poisson likelihood, y . f (y) = e = ey log log(y!) , y! Thus, = log , b( ) = , c(y, ) = log(y!), = 1, = e , b( ) = e , b ( ) = e = , 22/52. Link function is the parameter of interest, and needs to appear somehow in the likelihood function to use maximum likelihood. A link function g relates the Linear predictor X to the mean parameter , X = g( ). g is required to be monotone increasing and di erentiable = g 1 (X ). 23/52. Examples of link functions For LM, g( ) = identity. Poisson data. Suppose Y |X Poisson( (X)). (X) > 0.