Chapter 2: Maximum Likelihood Estimation - Accueil

1 Chapter 2: Maximum Likelihood EstimationAdvanced Econometrics - HEC LausanneChristophe HurlinUniversity of Orl ansDecember 9, 2013 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20131 / 207 Section 1 IntroductionChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20132 / 2071. IntroductionThe Maximum Likelihood Estimation (MLE) is a method ofestimating the parameters of a model. This Estimation method is oneof the most widely method of Maximum Likelihood selects the set of values of themodel parameters that maximizes the Likelihood function. Intuitively,this maximizes the "agreement" of the selected model with theobserved Maximum - Likelihood Estimation gives an uni ed approach Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20133 / 2072. The Principle of Maximum LikelihoodWhat are the main properties of the Maximum Likelihood estimator?

2 IIs it asymptotically unbiased?IIs it asymptotically e cient? Under which condition(s)?IIs it consistent?IWhat is the asymptotic distribution?How to apply the Maximum Likelihood principle to the multiple linearregression model, to the Probit/Logit Models etc. ?.. All of these questions are answered in this Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20134 / 2071. IntroductionThe outline of this Chapter is the following:Section 2:The principle of the Maximum Likelihood estimationSection 3:The Likelihood functionSection 4: Maximum Likelihood estimatorSection 5:Score, Hessian and Fisher informationSection 6:Properties of Maximum Likelihood estimatorsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20135 / 2071. IntroductionReferencesAmemiya T. (1985), Advanced Econometrics. Harvard University W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice HilPelgrin, F.

3 (2010), Lecture notes Advanced Econometrics, HEC Lausanne (aspecial thank)Ruud P., (2000) An introduction to Classical Econometric Theory, OxfordUniversity , E. (2001), Maximum Likelihood Estimation , Lecture Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20136 / 207 Section 2 The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20137 / 2072. The Principle of Maximum LikelihoodObjectivesIn this section, we present a simple example in order1To introduce thenotations2To introduce the notion introduce the concept ofmaximum Likelihood estimator4To introduce the concept ofmaximum Likelihood estimateChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20138 / 2072. The Principle of Maximum LikelihoodExampleSuppose that X1,X2, ,XNare discrete random variables, such thatXi Pois( )with apmf(probability mass function) de ned as:Pr(Xi=xi)=exp( ) xixi!

4 Where is an unknown parameter to Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20139 / 2072. The Principle of Maximum LikelihoodQuestion:What is the probability of observing theparticular samplefx1,x2,..,xNg, assuming that a Poisson distribution with as yet unknownparameter generated the data?This probability is equal toPr((X1=x1)\..\(XN=xN))Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201310 / 2072. The Principle of Maximum LikelihoodSince the this joint probability is equal to theproduct of the marginal probabilitiesPr((X1=x1)\..\(XN=xN))=N i=1Pr(Xi=xi)Given the pmf of the Poisson distribution, we have:Pr((X1=x1)\..\(XN=xN))=N i=1exp( ) xixi!=exp( N) Ni=1xiN i=1xi!Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201311 / 2072. The Principle of Maximum LikelihoodDe nitionThis joint probability is a function of (the unknown parameter) andcorresponds to thelikelihood of the samplefx1.

5 ,xNgdenoted byLN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))withLN( ; ,xN)=exp( N) N=1xi 1N i=1xi!Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201312 / 2072. The Principle of Maximum LikelihoodExampleLet us assume that forN=10,we have a realization of the sample equaltof5,0,1,1,0,3,2,3,4,1g,then:LN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))LN( ; ,xN)=e 10 20207,360 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201313 / 2072. The Principle of Maximum LikelihoodQuestion:What value of would make thissample most probable?Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201314 / 2072. The Principle of Maximum LikelihoodThis Figure plots the functionLN( ;x)for various values of . It has asingle mode at =2, which would be the Maximum Likelihood estimate,or MLE, of . 10-8 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201315 / 2072.

6 The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201316 / 2072. The Principle of Maximum LikelihoodConsider maximizing the Likelihood functionLN( ; ,xN)with respect to . Since the log function is monotonically increasing, we usually maximizelnLN( ; ,xN)instead. In this case:lnLN( ; ,xN)= N+ln( )N i=1xi ln N i=1xi! lnLN( ; ,xN) = N+1 N i=1xi 2lnLN( ; ,xN) 2= 1 2N i=1xi<0 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201317 / 2072. The Principle of Maximum LikelihoodUnder suitable regularity conditions, the Maximum Likelihood estimate(estimator) is de ned as:b =arg max 2R+lnLN( ; ,xN)FOC: lnLN( ; ,xN) b = N+1b N i=1xi=0()b =(1/N)N i=1xiSOC: 2lnLN( ; ,xN) 2 b = 1b 2N i=1xi<0b is a Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201318 / 2072. The Principle of Maximum LikelihoodThe Maximum likelihoodestimate(realization) is:b b (x)=1NN i=1xiGiven the samplef5,0,1,1,0,3,2,3,4,1g,we haveb (x)= Maximum likelihoodestimator(random variable) is:b =1NN i=1 XiChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201319 / 2072.

7 The Principle of Maximum LikelihoodContinuous variablesThe reference to the probability of observing the given sample is notexact in a continuous distribution, since a particular sample hasprobability zero. Nonetheless, the principle is the Likelihood function then corresponds to the pdf associated to thejoint distributionof(X1,X2,..,XN)evaluated at the point(x1,x2,..,xN):LN( ; ,xN)=fX1,..,XN(x1,x2,..,xN; )Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201320 / 2072. The Principle of Maximum LikelihoodContinuous variablesIf the random variablesfX1,X2,.., we have:LN( ; ,xN)=N i=1fX(xi; )wherefX(xi; )denotes the pdf of the marginal distribution ofX(orXisince all the variables have the same distribution).The values of the parameters that maximizeLN( ; ,xN)or its logare the Maximum Likelihood estimates, denotedb (x).Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201321 / 207 Section 3 The Likelihood functionDe nitions and NotationsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201322 / 2073.

8 The Likelihood FunctionObjectives1 Introduce thenotationsfor an Estimation problem that deals with amarginal distributionor aconditional distribution (model).2De ne thelikelihoodand the concept ofconditional log-likelihood4 Propose various applicationsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201323 / 2073. The Likelihood FunctionNotationsLet us consider a continuous random variableX,with a pdf denotedfX(x; ),forx2R =( K)|is aK 1 vector of unknown parameters. We assumethat 2 us consider a samplefX1,.., variables withthe same arbitrary distribution realisation offX1,..,XNg(the data ) is denotedfx1,..,xNgorxfor Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201324 / 2073. The Likelihood FunctionExample (Normal distribution)IfX N m, 2 then:fX(z; )=1 p2 exp (z m)22 2!8z2 RwithK=2 and = m 2 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201325 / 2073.

9 The Likelihood FunctionDe nition ( Likelihood Function)The Likelihood function is de ned to be:LN: RN!R+( ;x1,..,xn)7 !LN( ;x1,..,xn)=N i=1fX(xi; )Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201326 / 2073. The Likelihood FunctionDe nition (Log- Likelihood Function)The log- Likelihood function is de ned to be:`N: RN!R( ;x1,..,xn)7 !`N( ;x1,..,xn)=N i=1lnfX(xi; )Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201327 / 2073. The Likelihood FunctionRemark:the (log-) Likelihood function depends on two type of arguments:Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201328 / 2073. The Likelihood FunctionNotations:In the rest of the Chapter , I will use the following alternativenotations:LN( ;x) L( ;x1,..,xN) LN( )`N( ;x) lnLN( ;x) lnL( ;x1,..,xN) lnLN( )Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201329 / 2073.

10 The Likelihood FunctionExample (Sample of Normal Variables)We consider a samplefY1,.., m, 2 and denote therealisation byfy1,..,yNgory. Let us de ne = m 2 |,then we have:LN( ;y)=N i=11 p2 exp (yi m)22 2!= 22 N/2exp 12 2N i=1(yi m)2!`N( ;y)= N2ln 2 N2ln(2 ) 12 2N i=1(yi m)2 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201330 / 2073. The Likelihood FunctionDe nition ( Likelihood of one observation)We can also de ne the (log-) Likelihood ofone observationxi:Li( ;x)=fX(xi; )withLN( ;x)=N i=1Li( ;x)`i( ;x)=lnfX(xi; )with`N( ;x)=N i=1`i( ;x)Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201331 / 2073. The Likelihood FunctionExample (Exponential Distribution)Suppose thatD1,D2,.., random variables (durationsfor instance), withDi Exp( )with 0 andLi( ;di)=fD(di; )=1 exp di `i( ;di)=ln(fD(di; ))= ln( ) di Then we have:LN( ;d)= Nexp 1 N i=1di!