Chapter 2: Maximum Likelihood Estimation - Accueil

1 Chapter 2: Maximum Likelihood EstimationAdvanced econometrics - HEC LausanneChristophe HurlinUniversity of Orl ansDecember 9, 2013 Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20131 / 207 Section 1 IntroductionChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20132 / 2071. IntroductionThe Maximum Likelihood Estimation (MLE) is a method ofestimating the parameters of a model. This Estimation method is oneof the most widely method of Maximum Likelihood selects the set of values of themodel parameters that maximizes the Likelihood function.

2 Intuitively,this maximizes the "agreement" of the selected model with theobserved Maximum - Likelihood Estimation gives an uni ed approach Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20133 / 2072. The Principle of Maximum LikelihoodWhat are the main properties of the Maximum Likelihood estimator?IIs it asymptotically unbiased?IIs it asymptotically e cient? Under which condition(s)?IIs it consistent?IWhat is the asymptotic distribution?How to apply the Maximum Likelihood principle to the multiple linearregression model, to the Probit/Logit Models etc. ?.. All of these questions are answered in this Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20134 / 2071.

3 IntroductionThe outline of this Chapter is the following:Section 2:The principle of the Maximum Likelihood estimationSection 3:The Likelihood functionSection 4: Maximum Likelihood estimatorSection 5:Score, Hessian and Fisher informationSection 6:Properties of Maximum Likelihood estimatorsChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20135 / 2071. IntroductionReferencesAmemiya T. (1985), advanced econometrics . Harvard University W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice HilPelgrin, F. (2010), Lecture notes advanced econometrics , HEC Lausanne (aspecial thank)Ruud P.

4 , (2000) An introduction to Classical Econometric Theory, OxfordUniversity , E. (2001), Maximum Likelihood Estimation , Lecture Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20136 / 207 Section 2 The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20137 / 2072. The Principle of Maximum LikelihoodObjectivesIn this section, we present a simple example in order1To introduce thenotations2To introduce the notion introduce the concept ofmaximum Likelihood estimator4To introduce the concept ofmaximum Likelihood estimateChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20138 / 2072.

5 The Principle of Maximum LikelihoodExampleSuppose that X1,X2, ,XNare discrete random variables, such thatXi Pois( )with apmf(probability mass function) de ned as:Pr(Xi=xi)=exp( ) xixi!where is an unknown parameter to Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 20139 / 2072. The Principle of Maximum LikelihoodQuestion:What is the probability of observing theparticular samplefx1,x2,..,xNg, assuming that a Poisson distribution with as yet unknownparameter generated the data?This probability is equal toPr((X1=x1)\..\(XN=xN))Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201310 / 2072.

6 The Principle of Maximum LikelihoodSince the this joint probability is equal to theproduct of the marginal probabilitiesPr((X1=x1)\..\(XN=xN))=N i=1Pr(Xi=xi)Given the pmf of the Poisson distribution, we have:Pr((X1=x1)\..\(XN=xN))=N i=1exp( ) xixi!=exp( N) Ni=1xiN i=1xi!Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201311 / 2072. The Principle of Maximum LikelihoodDe nitionThis joint probability is a function of (the unknown parameter) andcorresponds to thelikelihood of the samplefx1,..,xNgdenoted byLN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))withLN( ; ,xN)=exp( N) N=1xi 1N i=1xi!

7 Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201312 / 2072. The Principle of Maximum LikelihoodExampleLet us assume that forN=10,we have a realization of the sample equaltof5,0,1,1,0,3,2,3,4,1g,then:LN( ; ,xN)=Pr((X1=x1)\..\(XN=xN))LN( ; ,xN)=e 10 20207,360 Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201313 / 2072. The Principle of Maximum LikelihoodQuestion:What value of would make thissample most probable?Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201314 / 2072. The Principle of Maximum LikelihoodThis Figure plots the functionLN( ;x)for various values of.

8 It has asingle mode at =2, which would be the Maximum Likelihood estimate,or MLE, of . 10-8 Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201315 / 2072. The Principle of Maximum LikelihoodChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201316 / 2072. The Principle of Maximum LikelihoodConsider maximizing the Likelihood functionLN( ; ,xN)with respect to . Since the log function is monotonically increasing, we usually maximizelnLN( ; ,xN)instead. In this case:lnLN( ; ,xN)= N+ln( )N i=1xi ln N i=1xi! lnLN( ; ,xN) = N+1 N i=1xi 2lnLN( ; ,xN) 2= 1 2N i=1xi<0 Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201317 / 2072.

9 The Principle of Maximum LikelihoodUnder suitable regularity conditions, the Maximum Likelihood estimate(estimator) is de ned as:b =arg max 2R+lnLN( ; ,xN)FOC: lnLN( ; ,xN) b = N+1b N i=1xi=0()b =(1/N)N i=1xiSOC: 2lnLN( ; ,xN) 2 b = 1b 2N i=1xi<0b is a Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201318 / 2072. The Principle of Maximum LikelihoodThe Maximum likelihoodestimate(realization) is:b b (x)=1NN i=1xiGiven the samplef5,0,1,1,0,3,2,3,4,1g,we haveb (x)= Maximum likelihoodestimator(random variable) is:b =1NN i=1 XiChristophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201319 / 2072.

10 The Principle of Maximum LikelihoodContinuous variablesThe reference to the probability of observing the given sample is notexact in a continuous distribution, since a particular sample hasprobability zero. Nonetheless, the principle is the Likelihood function then corresponds to the pdf associated to thejoint distributionof(X1,X2,..,XN)evaluated at the point(x1,x2,..,xN):LN( ; ,xN)=fX1,..,XN(x1,x2,..,xN; )Christophe Hurlin (University of Orl ans) advanced econometrics - HEC LausanneDecember 9, 201320 / 2072. The Principle of Maximum LikelihoodContinuous variablesIf the random variablesfX1,X2,.., we have:LN( ; ,xN)=N i=1fX(xi; )wherefX(xi; )denotes the pdf of the marginal distribution ofX(orXisince all the variables have the same distribution).