Chapter 2: Maximum Likelihood Estimation - univ …

1 Chapter 2: Maximum Likelihood EstimationAdvanced Econometrics - HEC LausanneChristophe HurlinUniversity of Orl ansDecember 9, 2013 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20131 / 207 Section 1 IntroductionChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20132 / 2071. IntroductionThe Maximum Likelihood Estimation (MLE) is a method ofestimating the parameters of a model. This Estimation method is oneof the most widely method of Maximum Likelihood selects the set of values of themodel parameters that maximizes the Likelihood function. Intuitively,this maximizes the "agreement" of the selected model with theobserved Maximum - Likelihood Estimation gives an uni ed approach Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20133 / 2072.

2 The Principle of Maximum LikelihoodWhat are the main properties of the Maximum Likelihood estimator?IIs it asymptotically unbiased?IIs it asymptotically e cient? Under which condition(s)?IIs it consistent?IWhat is the asymptotic distribution?How to apply the Maximum Likelihood principle to the multiple linearregression model, to the Probit/Logit Models etc. ?.. All of these questions are answered in this Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20134 / 2071. IntroductionThe outline of this Chapter is the following:Section 2:The principle of the Maximum Likelihood estimationSection 3:The Likelihood functionSection 4: Maximum Likelihood estimatorSection 5:Score, Hessian and Fisher informationSection 6:Properties of Maximum Likelihood estimatorsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20135 / 2071.

3 IntroductionReferencesAmemiya T. (1985), Advanced Econometrics. Harvard University W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice HilPelgrin, F. (2010), Lecture notes Advanced Econometrics, HEC Lausanne (aspecial thank)Ruud P., (2000) An introduction to Classical Econometric Theory, OxfordUniversity , E. (2001), Maximum Likelihood Estimation , Lecture Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20136 / 207 Section 2 The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20137 / 2072. The Principle of Maximum LikelihoodObjectivesIn this section, we present a simple example in order1To introduce thenotations2To introduce the notion introduce the concept ofmaximum Likelihood estimator4To introduce the concept ofmaximum Likelihood estimateChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20138 / 2072.

4 The Principle of Maximum LikelihoodExampleSuppose that X1,X2, ,XNare discrete random variables, such thatXi Pois( )with apmf(probability mass function) de ned as:Pr(Xi=xi)=exp( ) xixi!where is an unknown parameter to Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 20139 / 2072. The Principle of Maximum LikelihoodQuestion:What is the probability of observing theparticular samplefx1,x2,..,xNg, assuming that a Poisson distribution with as yet unknownparameter generated the data?This probability is equal toPr((X1=x1)\..\(XN=xN))Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201310 / 2072.

5 The Principle of Maximum LikelihoodSince the this joint probability is equal to theproduct of the marginal probabilitiesPr((X1=x1)\..\(XN=xN))=N i=1Pr(Xi=xi)Given the pmf of the Poisson distribution, we have:Pr((X1=x1)\..\(XN=xN))=N i=1exp( ) xixi!=exp( N) Ni=1xiN i=1xi!Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201311 / 2072. The Principle of Maximum LikelihoodDe nitionThis joint probability is a function of (the unknown parameter) andcorresponds to thelikelihood of the samplefx1,..,xNgdenoted byLN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))withLN( ; ,xN)=exp( N) N=1xi 1N i=1xi!Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201312 / 2072.

6 The Principle of Maximum LikelihoodExampleLet us assume that forN=10,we have a realization of the sample equaltof5,0,1,1,0,3,2,3,4,1g,then:LN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))LN( ; ,xN)=e 10 20207,360 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201313 / 2072. The Principle of Maximum LikelihoodQuestion:What value of would make thissample most probable?Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201314 / 2072. The Principle of Maximum LikelihoodThis Figure plots the functionLN( ;x)for various values of . It has asingle mode at =2, which would be the Maximum Likelihood estimate,or MLE, of.

7 10-8 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201315 / 2072. The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201316 / 2072. The Principle of Maximum LikelihoodConsider maximizing the Likelihood functionLN( ; ,xN)with respect to . Since the log function is monotonically increasing, we usually maximizelnLN( ; ,xN)instead. In this case:lnLN( ; ,xN)= N+ln( )N i=1xi ln N i=1xi! lnLN( ; ,xN) = N+1 N i=1xi 2lnLN( ; ,xN) 2= 1 2N i=1xi<0 Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201317 / 2072.

8 The Principle of Maximum LikelihoodUnder suitable regularity conditions, the Maximum Likelihood estimate(estimator) is de ned as:b =arg max 2R+lnLN( ; ,xN)FOC: lnLN( ; ,xN) b = N+1b N i=1xi=0()b =(1/N)N i=1xiSOC: 2lnLN( ; ,xN) 2 b = 1b 2N i=1xi<0b is a Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201318 / 2072. The Principle of Maximum LikelihoodThe Maximum likelihoodestimate(realization) is:b b (x)=1NN i=1xiGiven the samplef5,0,1,1,0,3,2,3,4,1g,we haveb (x)= Maximum likelihoodestimator(random variable) is:b =1NN i=1 XiChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201319 / 2072.

9 The Principle of Maximum LikelihoodContinuous variablesThe reference to the probability of observing the given sample is notexact in a continuous distribution, since a particular sample hasprobability zero. Nonetheless, the principle is the Likelihood function then corresponds to the pdf associated to thejoint distributionof(X1,X2,..,XN)evaluated at the point(x1,x2,..,xN):LN( ; ,xN)=fX1,..,XN(x1,x2,..,xN; )Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201320 / 2072. The Principle of Maximum LikelihoodContinuous variablesIf the random variablesfX1,X2,.., we have:LN( ; ,xN)=N i=1fX(xi; )wherefX(xi; )denotes the pdf of the marginal distribution ofX(orXisince all the variables have the same distribution).

10 The values of the parameters that maximizeLN( ; ,xN)or its logare the Maximum Likelihood estimates, denotedb (x).Christophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201321 / 207 Section 3 The Likelihood functionDe nitions and NotationsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201322 / 2073. The Likelihood FunctionObjectives1 Introduce thenotationsfor an Estimation problem that deals with amarginal distributionor aconditional distribution (model).2De ne thelikelihoodand the concept ofconditional log-likelihood4 Propose various applicationsChristophe Hurlin (University of Orl ans)Advanced Econometrics - HEC LausanneDecember 9, 201323 / 2073.