Lecture Notes 9 Asymptotic Theory (Chapter 9)

1 Lecture Notes 9. Asymptotic Theory (Chapter 9). In these Notes we look at the large sample properties of estimators, especially the maxi- mum likelihood estimator . Some Notation: Recall that Z. E (g(X)) g(x)p(x; )dx. 1 Review of o, O, etc. 1. an = o(1) mean an 0 as n . P. 2. A random sequence An is op (1) if An 0 as n . P. 3. A random sequence An is op (bn ) if An /bn 0 as n . P. 4. nb op (1) = op (nb ), so n op (1/ n) = op (1) 0. 5. op (1) op (1) = op (1). 6. an = O(1) if |an | is bounded by a constant as n . 7. A random sequence Yn is Op (1) if for every > 0 there exists a constant M such that limn P (|Yn | > M ) < as n.

2 8. A random sequence Yn is Op (bn ) if Yn /bn is Op (1). 9. If Yn Y , then Yn is Op (1).. 10. If n(Yn c) Y then Yn = OP (1/ n). 11. Op (1) Op (1) = Op (1). 12. op (1) Op (1) = op (1). 2 Distances Between Probability Distributions Let P and Q be distributions with densities p and q. We will use the following distances between P and Q. 1. Total variation distance TV(P, Q) = supA |P (A) Q(A)|. R. 2. L1 distance d1 (P, Q) = |p q|.. qR. 3. Hellinger distance h(P, Q) = ( p q)2 . R. 4. Kullback-Leibler distance K(P, Q) = p log(p/q).

3 1. R. 5. L2 distance d2 (P, Q) = (p q)2 . Here are some properties of these distances: 1. TV(P, Q) = 12 d1 (P, Q). (prove this!). R . 2. h2 (P, Q) = 2(1 pq). p 3. TV(P, Q) h(P, Q) 2TV(P, Q). 4. h2 (P, Q) K(P, Q). p 5. TV(P, Q) h(P, Q) K(P, Q). p 6. TV(P, Q) K(P, Q)/2. 3 Consistency An estimator bn = g(X1 , .. , Xn ) is consistent for if P. bn . as n . In other words, bn = op (1). Here are two common ways to prove that bn consistent. Method 1: Show that, for all > 0, P(| bn | ) 0. Method 2. Prove convergence in quadratic mean: MSE( bn ) = Bias2 ( bn ) + Var( bn ) 0.

4 Qm p If bias 0 and var 0 then bn which implies that bn . Example 1 X1 , .. , Xn Bernoulli(p). The mle pb has bias 0 and variance p(1 p)/n 0. P P. Here pb = i Xi /n. So pb p and is consistent. Now let = log(p/(1 p)). Then p/(1 pb)). Now b = g(b b = log(b p) where g(p) = log(p/(1 p)). By the continuous mapping P. theorem, so this is consistent. Now consider b P. Xi + 1. pb = i . n+1. 2. Then p 1. p) p = . Bias = E(b 0. n(1 + n). and p(1 p). Var = 0. n So this is consistent. Example 2 X1 , .. , Xn Uniform(0, ). Let bn = X(n).

5 By direct proof (we did it earlier). P. we have bn . Method of moments estimators are typically consistent. Consider one parameter. Recall b = m where m = n 1 Pn Xi . Assume that 1 exists and is continuous. So that ( ) i=1. 1 P. = (m). By the WLLN m ( ). So, by the continuous mapping Theorem, b P. bn = 1 (m) 1 ( ( )) = . 4 Consistency of the MLE. Under regularity conditions, the mle is consistent. Let us prove this in a special case. This will also reveal a connection between the mle and Hellinger distance. Suppose that the model consists of finitely many distinct densities P = {p0 , p1.}

6 , pN }. The likelihood function is n Y. L(pj ) = pj (Xi ). i=1. The mle pb is the density pj that maximizes L(pj ). Without loss of generality, assume that the true density is p0 . Theorem 3. p 6= p0 ) 0. P(b as n . Proof. Let us begin by first proving an inequality. Let j = h(p0 , pj ). Then, for j 6= 0, ! s ! n n L(pj ) 2. Y p j (X i ) 2. Y p j (X i ) 4. P > e n j /2 = P > e n j /2 = P > e n j /2. L(p0 ) i=1 0. p (X i ) i=1. p 0 (X i ). s ! s ! n n n 2j /4. Y pj (Xi ) n 2j /4. Y pj (Xi ). e E =e E. i=1. p0 (Xi ) i=1. p0 (Xi ).

7 N n n h2 (p0 , pj ) 2j Z . n 2j /4 n 2j /4 n 2j /4. = e pj p0 =e 1 =e 1 . 2 2. 2 . j . 2 2 2 2. = en j /4 exp n log 1 en j /4 e n j /2 = e n j /2 . 2. 3. R . We used the fact that h2 (p0 , pj ) = 2 2 p0 pj and also that log(1 x) x for x > 0. Let = min{ 1 , .. , N }. Then . L(pj ) n 2j /2. P(bp 6= p0 ) P >e for some j L(p0 ). N . X L(pj ) n 2j /2. P >e j=1. L(p0 ). N. 2 2 /2. X. e n j /2 N e n 0. j=1.. We can prove a similar result using Kullback-Leibler distance as follows. Let X1 , X2 , .. be iid F . Let 0 be the true value of and let be some other value.

8 We will show that L( 0 )/L( ) > 1 with probability tending to 1. We assume that the model is identifiable;. this means that 1 6= 2 implies that K( 1 , 2 ) > 0 where K is the Kullback-Leibler distance. Theorem 4 Suppose the model is identifiable. Let 0 be the true value of the parameter. For any 6= 0 . L( 0 ). P >1 1. L( ). as n . Proof. We have n n 1 1X 1X. (`( 0 ) `( )) = log p(Xi ; 0 ) log p(Xi ; ). n n i=1 n i=1. p E(log p(X; 0 )) E(log p(X; )). Z Z. = (log p(x; 0 ))p(x; 0 )dx (log p(x; ))p(x; 0 )dx Z . p(x; 0 ). = log p(x; 0 )dx p(x; ).

9 = K( 0 , ) > 0. So . L( 0 ). P >1 = P (`( 0 ) `( ) > 0). L( ).. 1. = P (`( 0 ) `( )) > 0 1.. n . This is not quite enough to show that bn 0 . 4. Example 5 Inconsistency of an mle. In all examples so far n , but the number of parameters is fixed. What if the number of parameters also goes to ? Let Y11 , Y12 N ( 1 , 2 ). Y21 , Y22 N ( 2 , 2 ).. Yn1 , Yn2 N ( n , 2 ). Some calculations show that n X 2. 2. X (Yij Y i )2.. b = . i=1 j=1. 2n It is easy to show (good test question) that p 2. b2 .. 2. 2 is consistent. Note that the modified estimator 2b The reason why consistency fails is because the dimension of the parameter space is increasing with n.

10 Theorem 6 Under regularity conditions on the model {p(x; ) : }, the mle is consis- tent. The regularity conditions are technical. Basically, we need to assume that (i) the dimen- sion of the parameter space does not change with n and (ii) p(x; ) is a smooth function of . 5 Score and Fisher Information The score function and Fisher information are the key quantities in many aspects of statis- tical inference. Suppose for now that R. The score function is log p(X1 , .. , Xn ; ) iid X log p(Xi ; ). Sn ( ) Sn ( , X1 , .. , Xn ) = `0 ( ) = =.