Chapter 5 The Delta Method and Applications

1 Chapter 5 The Delta Method and Linear approximations of functionsIn the simplest form of the central limit theorem, Theorem , we consider a sequenceX1,X2,..of independent and identically distributed (univariate) random variables withfinite variance 2. In this case, the central limit theorem states that n(Xn )d Z,( )where = EX1andZis a standard normal random this Chapter , we wish to consider the asymptotic distribution of, say, some function ofXn. In the simplest case, the answer depends on results already known: Consider a linearfunctiong(t) =at+bfor some known constantsaandb. Since EXn= , clearly Eg(Xn) =a +b=g( ) by the linearity of the expectation operator. Therefore, it is reasonable to askwhether n[g(Xn) g( )] tends to some distribution asn . But the linearity ofg(t)allows one to write n[g(Xn) g( )]=a n(Xn ).We conclude by Theorem that n[g(Xn) g( )]d a course, the distribution on the right hand side above isN(0,a2 2).None of the preceding development is especially deep; one might even say that it is obviousthat a linear transformation of the random variableXnalters its asymptotic distribution85by a constant multiple.

2 Yet what if the functiong(t) is nonlinear? It is in this nonlinearcase that a strong understanding of the argument above, as simple as it may be, pays realdividends. For ifXnis consistent for (say), then we know that, roughly speaking,Xnwill be very close to for largen. Therefore, the only meaningful aspect of the behavior ofg(t) is its behavior in a small neighborhood of .And in a small neighborhood of ,g( )may be considered to be roughly a linear function if we use a first-order Taylor particular, we may approximateg(t) g( ) +g ( )(t )fortin a small neighborhood of . We see thatg ( ) is the multiple oft, and so the logic ofthe linear case above suggests n{g(Xn) g( )}d g ( ) 2Z.( )Indeed, expression ( ) is a special case of the powerful theorem known as the Delta Method :Ifg (a) exists andnb(Xn a)d Xforb>0, thennb{g(Xn) g(a)}d g (a) proof of the Delta Method uses Taylor s theorem, Theorem : SinceXn aP 0,nb{g(Xn) g(a)}=nb(Xn a){g (a) +oP(1)},and thus Slutsky s theorem together with the fact thatnb(Xn a)d Xproves the ( ) may be reexpressed as a corollary of Theorem :Corollary often-used special case of Theorem in whichXis normallydistributed states that ifg ( ) exists and n(Xn )d N(0, 2), then n{g(Xn) g( )}d N{0, 2g ( )2}.

3 Ultimately, we will extend Theorem in two directions: Theorem deals with the specialcase in whichg (a) = 0, and Theorem is the multivariate version of the Delta Method . Butwe first apply the Delta Method to a couple of simple examples that illustrate a frequentlyunderstood but seldom stated principle: When we speak of the asymptotic distribution ofa sequence of random variables, we generally refer to a nontrivial ( , nonconstant) distribu-tion. Thus, in the case of an independent and identically distributed sequenceX1,X2,..ofrandom variables with finite variance, the phrase asymptotic distribution ofXn generallyrefers to the fact that n(Xn EX1)d N(0,VarX1),notthe fact thatXnP distribution ofX2nSupposeX1,X2,..are iid with mean and finite variance 2. Then by the central limit theorem, n(Xn )d N(0, 2).Therefore, the Delta Method gives n(X2n 2)d N(0,4 2 2).( )However, this is not necessarily the end of the story. If = 0, then the normallimit in ( ) is degenerate that is, expression ( ) merely states that n(X2n)converges in probability to the constant 0.

4 This is not what we mean by theasymptotic distribution! Thus, we must treat the case = 0 separately, notingin that case that nXnd N(0, 2) by the central limit theorem, which impliesthatnXnd 2 binomial variance:SupposeXn binomial(n,p). BecauseXn/nis the maximum likelihood estimator forp, the maximum likelihood esti-mator forp(1 p) is n=Xn(n Xn)/n2. The central limit theorem tells us that n(Xn/n p)d N{0,p(1 p)}, so the Delta Method gives n{ n p(1 p)}d N{0,p(1 p)(1 2p)2}.Note that in the casep= 1/2, this does not give the asymptotic distribution of n. Exercise gives a hint about how to find the asymptotic distribution of nin this have seen in the preceding examples that ifg (a) = 0, then the Delta Method givessomething other than the asymptotic distribution we seek. However, by using more termsin the Taylor expansion, we obtain the following generalization of Theorem :Theorem (t) hasrderivatives at the pointaandg (a) =g (a) = =g(r 1)(a) = 0, thennb(Xn a)d Xforb>0 implies thatnrb{g(Xn) g(a)}d 1r!

5 G(r)(a) is straightforward using the multivariate notion of differentiability discussed in to prove the following theorem:87 Theorem Delta Method :Ifg:Rk R`has a derivative g(a) ata Rkandnb(Xn a)d Yfor somek-vectorYand some sequenceX1,X2,..ofk-vectors, whereb>0,thennb{g(Xn) g(a)}d [ g(a)] proof of Theorem involves a simple application of the multivariate Taylor expansionof Equation ( ).Exercises for Section nbe defined as in Example Find the asymptotic distributionof nin the casep= 1/2. That is, find constant sequencesanandbnand anontrivial random variableXsuch thatan( n bn)d :LetYn=Xn (n/2). Apply the central limit theorem toYn, thentransform both sides of the resulting limit statement so that a statement involving Theorem Variance stabilizing transformationsOften, if E (Xi) = is the parameter of interest, the central limit theorem gives n(Xn )d N{0, 2( )}.In other words, the variance of the limiting distribution is a function of.

6 This is a problemif we wish to do inference for , because ideally the limiting distribution should not dependon the unknown . The Delta Method gives a possible solution: Since n{g(Xn) g( )}d N{0, 2( )g ( )2},we may search for a transformationg(x) such thatg ( ) ( ) is a constant. Such a transfor-mation is called a variance stabilizing thatX1,X2,..are independent normal random variableswith mean 0 and variance 2. Let us define 2= VarX2i, which for the normaldistribution may be seen to be 2 4. (To verify this, try showing that EX4i= 3 4by differentiating the normal characteristic function four times and evaluating atzero.) Thus, Example shows that n(1nn i=1X2i 2)d N(0,2 4).To do inference for 2when we believe that our data are truly independent andidentically normally distributed, it would be helpful if the limiting distributiondid not depend on the unknown 2. Therefore, it is sensible in light of to search for a functiong(t) such that 2[g ( 2)]2 4is not a function of 2.

7 Inother words, we wantg (t) to be proportional to t 2=|t| 1. Clearlyg(t) = logtis such a function. Therefore, we call the logarithm function a variance-stabilizingfunction in this example, and Corollary shows that n{log(1nn i=1X2i) log( 2)}d N(0,2).Exercises for Section binomial(n,p), where 0<p<1.(a)Find the asymptotic distribution ofg(Xn/n) g(p), whereg(x) = min{x,1 x}.(b)Show thath(x) = sin 1( x) is a variance-stabilizing :(d/du) sin 1(u) = 1/ 1 ,X2,..be independent fromN( , 2) where 6= 0. LetS2n=1nn i=1(Xi Xn) the asymptotic distribution of the coefficient of binomial(n,p), wherep (0,1) is unknown. Obtain confi-dence intervals forpin two different ways:89(a)Since n(Xn/n p)d N[0,p(1 p)], the variance of the limiting distributiondepends only onp. Use the fact thatXn/nP pto find a consistent estimator ofthe variance and use it to derive a 95% confidence interval forp.(b)Use the result of problem (b) to derive a 95% confidence interval forp.

8 (c)Evaluate the two confidence intervals in parts (a) and (b) numerically forall combinations ofn {10,100,1000}andp {.1,.3,.5}as follows: For 1000realizations ofX bin(n,p), construct both 95% confidence intervals and keeptrack of how many times (out of 1000) that the confidence intervals the observed proportion of successes for each (n,p) combination. Doesyour study reveal any differences in the performance of these two competingmethods? Sample MomentsThe weak law of large numbers tells us that IfX1,X2,..are independent and identicallydistributed with E|X1|k< , then1nn i=1 XkiP is, sample moments are (weakly) consistent. For example, the sample variance, whichwe define ass2n=1nn i=1(Xi Xn)2=1nn i=1X2i (Xn)2,( )is consistent for VarXi= EX2i (EXi) , consistency is not the end of the story. The central limit theorem and the deltamethod will prove very useful in deriving asymptotic distribution results about sample mo-ments. We consider two very important examples involving the sample variance of Equation( ).

9 Example of sample T statistic:SupposeX1,X2,..are iid withE (Xi) = and Var (Xi) = 2< . Defines2nas in Equation ( ), and letTn= n(Xn ) n(Xn ) andBn= /sn, we obtainTn=AnBn. Therefore, sinceAnd N(0,1) by thecentral limit theorem andBnP 1 by the weak law of large numbers, Slutsky stheorem implies thatTnd N(0,1). In other words, T statistics are asymptoticallynormal under the null ,X2,..be independent and identically distributed with mean , variance 2, third central moment E (Xi )3= , and Var (Xi )2= 2< . DefineS2nas in Equation ( ). We have shown earlier that n(S2n 2)d N(0, 2). The same fact may be proven using Theorem as , letYi=Xi andZi=Y2i. We may use the multivariate central limittheorem to find the joint asymptotic distribution ofYnandZn, namely n{(YnZn) (0 2)}d N2{0,( 2 2)}.Note that the above result uses the fact that Cov (Y1,Z1) = .We may writeS2n=Zn (Yn)2. Therefore, define the functiong(a,b) =b a2and note that this gives g(a,b) = ( 2a,1).

10 To use the Delta Method , we shouldevaluate g(0, 2)( 2 2) g(0, 2)T= (0,1)( 2 2)(01)= g(0, 2)T= 2We conclude that n{g(YnZn) g(0 2)}= n(S2n 2)d N(0, 2)as we found earlier (using a different argument) in Example for Section thatX1,X2,..are iid Normal (0, 2) random variables.(a)Based on the result of Example , Give an approximate test at =.05 forH0: 2= : 26= (b)Forn= 25, estimate the true level of the test in part (a) for 20= 1 bysimulating 5000 samples of sizen= 25 from the null distribution. Report theproportion of cases in which you reject the null hypothesis according to your test(ideally, this proportion will be about .05). Sample CorrelationSuppose that (X1,Y1),(X2,Y2),..are iid vectors with EX4i< and EY4i< . Forthe sake of simplicity, we will assume without loss of generality that EXi= EYi= 0(alternatively, we could base all of the following derivations on the centered versions of therandom variables).We wish to find the asymptotic distribution of the sample correlation coefficient,r.