A Generalized Linear Model for Bernoulli Response Data

1 A Generalized Linear Model forBernoulli Response DataCopyrightc 2017 Dan Nettleton (Iowa State University)Statistics 5101 / 46 Consider the Gauss-Markov Linear Model with normalerrors:y=X + , N(0, 2I).Another way to write this Model is8i=1,..,n,yi N( i, 2), i=x0i ,andy1,..,ynare 2017 Dan Nettleton (Iowa State University)Statistics 5102 / 46 This is a special case of what is known as ageneralized Linear is another special case:8i=1,..,n,yi Bernoulli ( i), i=exp(x0i )1+exp(x0i ),andy1,..,ynare 2017 Dan Nettleton (Iowa State University)Statistics 5103 / 46In each example, all responses are independent, andeach Response is a draw from one type of distributionwhose parameters may depend on explanatoryvariables through a Linear predictorx0i .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 5104 / 46 The second Model , for the case of a binary Response ,is often called a logistic regression responses are common (success/failure,survive/die, good customer/bad customer, win/lose,etc.)

2 The logistic regression Model can help us understandhow explanatory variables are related to theprobability of success. Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 5105 / 46 Example: Disease Outbreak StudySource:Applied Linear Statistical Models, 4th edition,by Neter, Kutner, Nachtsheim, Wasserman (1996)In a health study to investigate an epidemic outbreakof a disease that is spread by mosquitoes, individualswere randomly sampled within two sectors in a city todetermine if the person had recently contracted thedisease under 2017 Dan Nettleton (Iowa State University)Statistics 5106 / 46 Response Variableyi=0(personidoes not have the disease)yi=1(personihas the disease)Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 5107 / 46 Potential Explanatory Variablesage in yearssocioeconomic status1=upper2=middle3=lowersector (1 or 2)Copyrightc 2017 Dan Nettleton (Iowa State University)

3 Statistics 5108 / 46 Questions of InterestThe potential explanatory variables and the responsewere recorded for 196 randomly selected any of these variables associated with theprobability of disease and if so how?Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 5109 / 46We will demonstrate how to useRto fit a logisticregression Model to this delving more deeply into logistic regression,we will review the basic facts of the 2017 Dan Nettleton (Iowa State University)Statistics 51010 / 46y Bernoulli ( )has probability mass functionPr(y=k)=f(k)=( k(1 )1 kfork2{0,1}0otherwiseThus,Pr(y=0)=f(0)= 0(1 )1 0=1 andPr(y=1)=f(1)= 1(1 )1 1= .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51011 / 46 The variance ofyis a function of the mean (y)=1Xk=0kf(k)=0 (1 )+1 = E(y2)=1Xk=0k2f(k)=02 (1 )+12 = Var(y)=E(y2) [E(y)]2= 2= (1 )Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51012 / 46 The Logistic Regression ModelFori=1.)

4 ,n,yi Bernoulli ( i),where i=exp(x0i )1+exp(x0i )andy1,..,ynare 2017 Dan Nettleton (Iowa State University)Statistics 51013 / 46 The Logit FunctionThe functiong( )=log 1 is called thelogit logit function maps the interval(0,1)to the realline( 1,1). is a probability, solog( 1 )is the log(odds), wherethe odds of an eventA Pr(A)1 Pr(A).Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51014 / 46 Note thatg( i)=log i1 i =log exp(x0i )1+exp(x0i ) 11+exp(x0i ) =log[exp(x0i )]=x0i .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51015 / 46 Thus, the logistic regression Model says that,yi Bernoulli ( i),wherelog i1 i =x0i In Generalized Linear Models terminology, the logit iscalled the link function because it links the mean ofyi( , i) to the Linear predictorx0i .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51016 / 46 For Generalized Linear Models, it is not necessarythat the mean ofyibe a Linear function of.

5 Rather, some function of the mean ofyiis a linearfunction of .For logistic regression, that function islogit( i)=log i1 i =x0i .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51017 / 46 When the Response is Bernoulli or more generally,binomial, the logit link function is one natural , other link functions can be common choices (that are also available inR)include the following:Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51018 / 4611logit:log 1 =x0 .2probit: 1( )=x0 ,where 1( )is the inverse ofN(0,1) log-log (cloglog in R):log( log(1 ))=x0 .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51019 / 46 Although any of these link functions (or others) canbe used, the logit link has some advantages when itcomes to interpreting the results (as we will discusslater).

6 Thus, the logit link is a good choice if it can provide agood fit to the 2017 Dan Nettleton (Iowa State University)Statistics 51020 / 46 The log likelihood function for logistic regression is`( |y)=nXi=1log[ yii(1 i)1 yi]=nXi=1[yilog( i)+(1 yi)log(1 i)]=nXi=1[yi{log( i) log(1 i)}+log(1 i)]=nXi=1 yilog i1 i +log(1 i) =nXi=1[yix0i log(1+exp{x0i })]Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51021 / 46 For Generalized Linear Models, Fisher s ScoringMethod is typically used to obtain an MLE for ,denoted as .Fisher s Scoring Method is a variation of theNewton-Raphson algorithm in which the Hessianmatrix (matrix of second partial derivatives) isreplaced by its expected value (-Fisher Informationmatrix).Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51022 / 46 For Generalized Linear Models, Fisher s scoringmethod results in an iterative weighted least algorithm is presented for the general case inSection ofGeneralized Linear Models2nd Edition(1989) by McCullough and 2017 Dan Nettleton (Iowa State University)Statistics 51023 / 46 For sufficiently large samples, is approximatelynormal with mean and a variance-covariancematrix that can be approximated by the estimatedinverse of the Fisher Information Matrix.

7 , N( , I 1( ))Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51024 / 46 Inference can be conducted using the Wald approachor via likelihood ratio testing as discussed in ourcourse notes on likelihood-related example, a Wald confidence interval forc0 withapproximate coverage probability given byc0 I 1( )cCopyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51025 / 46 Interpretation of Logistic Regression ParametersLetx=[x1,x2,..,xj 1,xj,xj+1,..,xp] x=[x1,x2,..,xj 1,xj+1,xj+1,..,xp] other words, xis the same asxexcept that thejthexplanatory variable has been increased by one =exp(x0 )1+exp(x0 )and =exp( x0 )1+exp( x0 ).Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51026 / 46 The Odds Ratio 1 1 =exp log 1 1 =exp log 1 log 1 =exp{ x0 x0 }=exp{(xj+1) j xj j}=exp{ j}.

8 Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51027 / 46 Thus, 1 =exp( j) 1 .All other explanatory variables held constant, theodds of success atxj+1areexp( j)times the odds ofsuccess is true regardless of the initial 2017 Dan Nettleton (Iowa State University)Statistics 51028 / 46A one unit increase in thejth explanatory variable(with all other explanatory variables held constant) isassociated with a multiplicative change in the odds ofsuccess by the factorexp( j).Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51029 / 46If(Lj,Uj)is a100(1 )%confidence interval for j,then(exp(Lj),exp(Uj))is a100(1 )%confidence interval forexp( j).Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51030 / 46P(L;Ep;EUj)=100(i-a)%#P(exp(L;)Eexp(p; )Eexp(U;))=100It a)%Also, note that =exp(x0 )1+exp(x0 )=11exp(x0 )+1=11+exp( x0 ).

9 Thus, if (Lj,Uj) is a100(1 )%confidence interval forx0 , then a100(1 )%confidence interval for is 11+exp( Lj),11+exp( Uj) .Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51031 / 46> d= (" ")> head(d)id age ses sector disease savings1 1 33 1 1 0 12 2 35 1 1 0 133 6 1 1 0 04 4 60 1 1 0 15 5 18 3 1 1 06 6 26 3 1 0 0>> d$ses=factor(d$ses)> d$sector=factor(d$sector)Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51032 / 46 LwtIanortTHISVARIABLEINTHISEXAMPLE.> o=glm(disease age+ses+sector,+ family=binomial(link=logit),+ data=d)>> summary(o)Call:glm(formula = disease age + ses + sector,family = binomial(link = logit),data = d)Deviance Residuals:Min 1Q Median 3Q 2017 Dan Nettleton (Iowa State University)Statistics 51033 / 46X=[1,aze,SEI,se=3,Sester2) :Estimate Std.]

10 Error z value Pr(>|z|)(Intercept) **age **ses2 **---Signif. codes: 0** ** * . 1(Dispersion parameter for binomial family taken to be 1)Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51034 / Test deviance: on 195 degrees of freedomResidual deviance: on 191 degrees of freedomAIC: of Fisher Scoring iterations: 3 Copyrightc 2017 Dan Nettleton (Iowa State University)Statistics 51035 / 46 NULLM oser:logit(Hi)=M(IT,=Tz= )FuuMorsel:logit(Hi)= coef(o)(Intercept) age ses2 ses3 > round(vcov(o),3)(Intercept) age ses2 ses3 sector2(Intercept) 2017 Dan Nettleton (Iowa State University)Statistics 51036 / 46 'B'B'3$4B'sVEr(E)bees> confint(o)Waiting for profiling to be % %(Intercept) 2017 Dan Nettleton (Iowa State University)Statistics 51037 / 46{bs:Zech-2llEb'D)extra}"*""" 1 " % r% )(,)<APPROXIMATE95%CONFIDENCEINTERVALFor Be.