Convergence in Distribution Central Limit Theorem

1 Convergence in DistributionCentral Limit TheoremStatistics 110 Summer 2006 Copyrightc 2006 by Mark E. IrwinConvergence in Bin(n, p)and let =np, Thenlimn P[X=x] = limn (nx)px(1 p)n x=e xx!So whenngets large, we can approximate binomial probabilities withPoisson (nx)px(1 p)n x= limn (nx)( n)x(1 n)n x=n!x!(n x)! x(1nx)(1 n) x(1 n)nConvergence in Distribution1=n!x!(n x)! x(1nx)(1 n) x(1 n)n= xx!limn n!(n x)!1(n )x 1(1 n)n e =e xx!2 Note that approximation works better whennis large andpis small ascan been seen in the following plot. Ifpis relatively large, a differentapproximation should be used. This is coming later.(Note in the plot, bars correspond to the true binomial probabilities and thered circles correspond to the Poisson approximation.) Convergence in Distribution201234lambda = 1 n = 10 p = (x) = 1 n = 50 p = (x) = 1 n = 200 p = (x) 1 2 3 4 5 6 7 8 9lambda = 5 n = 10 p = (x) = 5 n = 50 p = (x) = 5 n = 200 p = (x) in Distribution3 Example: LetY1, Y2.

2 Be iidExp(1). ThenXn=Y1+Y2+..+Yn Gamma(n,1)which hasE[Xn] =n; Var(Xn) =n;SD(Xn) = nThusZn=Xn n nhas mean = 0 and variance = compare its Distribution toZ N(0,1). IsP[ 1 Zn 2] P[ 1 Z 2]?LetZn=Xn n n;Xn=n+ nZnfZn(z) =fXn(n+ nz) nConvergence in Distribution4P[a Zn b] = bafZn(z)dz= ba nfXn(n+ nz)dz= ba n(n+ nz)n 1(n 1)!e (n+ nz)dzTo go further we need Stirling s Formula:n! nne n 2 n. SofXn(n+ nz) n=e n z n(n+z n)n 1 n(n 1)! e n z n(n+z n)n 1 n(n 1)n 1e n+1 2 n 1 2 e z n(1 +z n)n gn(z) Convergence in Distribution5log(gn(z)) = z n+nlog(1 +z n)= z n+n[z n 12z2n+13z3n3/2 ..] 12z2+O(1 n)sofXn(n+z n) n 1 2 e z2/2 ThusP[a Zn b] ba1 2 e z2/2dz=P[a Z b]So asnincreases, the Distribution ofZngets closer and closer to aN(0,1). Convergence in Distribution6 Another way of thinking of this, is that the Distribution ofXn=n+Zn napproaches that of aN(n, n).

3 = 2xf(x) 20246810 = 5xf(x) = 10xf(x) = 20xf(x) = 50xf(x)708090100 110 120 = 100xf(x) Convergence in , X2, ..be a sequence of RVs with cumulativedistribution functionsF1, F2, ..and letXbe a RV with distributionF. We sayXnConverges in DistributiontoXiflimn Fn(x) =F(x)at every point at whichFis XAn equivalent statement to this is that for allaandbwhereFis continuousP[a Xn b] P[a X b]Note that ifXnandXare discrete distributions, this condition reduces toP[Xn=xi] P[X=xi]for all support in Distribution8 Note that an equivalent definition of Convergence in Distribution is thatXnD XifE[g(Xn)] E[g(X)]for all bounded, continuous functionsg( ).This statement of Convergence in Distribution is needed to help prove thefollowing theoremTheorem. [Continuity Theorem ]LetXnbe a sequence of randomvariables with cumulative Distribution functionsFn(x)and correspondingmoment generating functionsMn(t).

4 LetXbe a random variable withcumulative Distribution functionF(x)and moment generating functionM(t). IfMn(t) M(t)for alltin an open interval containing zero, thenFn(x) F(x)at all continuity points ofF. That isXnD the previous two examples (Binomial/Poisson and Gamma/Normal)could be proved this in Distribution9 For the Gamma/Normal exampleMZn(t) =MXn(t n)e t n=(11 t n)ne t nSimilarly to the earlier proof, its easier to work withlogMZn(t)logMZn(t) = t n nlog(1 t n)= t n n[ t n 12t2n 13t3n3/2 ..]=12t2+O(1 n)ThusMZn(t) et2/2which is the MGF for a standard in Distribution10 Central Limit TheoremTheorem. [ Central Limit Theorem (CLT)]LetX1, X2, X3, ..be asequence of independent RVs having mean and variance 2and acommon Distribution functionF(x)and moment generating functionM(t)defined in a neighbourhood of zero. LetSn=n i=1 XnThenlimn P[Sn n n x]= (x)That isSn n nD N(0,1) Central Limit a loss of generality, we can assume that = 0.

5 So letZn=Sn n. SinceSnis the sum ofniid RVs,MSn(t) = (M(t))n;MZn(t) =(M(t n))nTaking a Taylor series expansion ofM(t)around 0 givesM(t) =M(0) +M (0)t+12M (0)t2+ t= 1 +12 2t2+O(t3)sinceM(0) = 1, M (0) = = 0, M (0) = 2. SoM(t n)= 1 +12 2(t n)2+O((t n)3)= 1 +t22n+O(1n3/2) Central Limit Theorem12 This givesMZn(t) =(1 +t22n+O(1n3/2))n et2/22 Note that the requirement of a MGF is not needed for the Theorem to fact, all that is needed is thatVar(Xi) = 2< . A standard proof ofthis more general Theorem uses the characteristic function (which is definedfor any Distribution ) (t) = eitxf(x)dx=M(it)instead of the moment generating functionM(t), wherei= the CLT holds for distributions such as the log normal, even thoughit doesn t have a Limit Theorem13 Also, the CLT is often presented in the following equivalent formZn= Xn / n= n Xn D N(0,1)To see this is the same, just multiply the numerator and denominator bynin the first form to get the statement common way that this is used is thatSnapprox.

6 N(n , n 2)or Xnapprox. N( , 2n) Central Limit Theorem14 Example: Insurance claimsSuppose that an insurance company has 10,000 policy holders. The expectedyearly claim per policyholder is $240 with a standard deviation of $ is the approximate probability that the total yearly claimsS10,000>$ MillionE[S10,000] = 10,000 240 = 2,400,000SD(S10,000) = 10,000 800 = 80,000P[S10,000>2,600,000]=P[S10,000 2,400,00080,000>2,600,000 2,400,00080,000] P[Z > ] = that this probability statement does not use anything about thedistribution of the original policy claims except their mean and standarddeviation. Its probable that their Distribution is highly skewed right (since x<< x), but the calculations ignore this Limit Theorem15 One consequence of the CLT is the normal approximation to the Bin(n, p)and pn=Xnn, then (sinceXncan be thought of the sumofnBernoulli s)Xn np np(1 p)D N(0,1); pn p p(1 p)/nD N(0,1)Another way of think of this is thatXnapprox.

7 N(np, np(1 p)); pnapprox. N(p,p(1 p)n)This approximation works better whenpis closer to12than whenpis near0 or rule of thumb is that is ok to use the normal approximation whennp 5andn(1 p) 5(expect at least 5 successes and 5 failures). (Other bookssometimes suggest other values, with the most popular alternative being10.) Central Limit Theorem1601234567p = n = 10xp(x) 5 7 912151821p = n = 50xp(x) 36 41 46 51 56 61 66p = n = 200xp(x) 1 2 3 4 5 6 7 8 9p = n = 10xp(x) 17 20 23 26 29 32 35p = n = 50xp(x) 84 90 96 103 111 119p = n = 200xp(x) Limit Theorem17012p = n = 10xp(x) = n = 100xp(x) 2 4 6 811 14 17 20p = n = 1000xp(x) = n = 10xp(x) 3 5 7 9121518p = n = 100xp(x) 79 87 95 104 114 124p = n = 1000xp(x) Limit Theorem18 Continuity correction to the binomial approximationp = n = = n = thatX Bin(50, )

8 And we are interested inP[ p ] =P[X 12]Notice that the bar correspondingtoX= 12, the normal curve onlypicks up about half the area, asthe bar actually goes from normal approximation can beimproved if we ask for the areaunder the normal curve up Limit Theorem19 LetY N(15, )(approximating normal). ThenP[X 12] = (True Probability)P[Y 12] = (No correction)P[Y ] = (With correction)p = n = Limit Theorem20 While this does give a better answer for many problems, normally Irecommend ignoring it. If the correction makes a difference, you probablywant to be doing an exact probability calculation will the CLT work better? Bign Distribution ofXiclose to normal. Approximation holds exactly ifn= 1ifXi N( , 2). Xiroughly symmetric. As we observed with the binomial examples, thecloserpwas to , thus closer to symmetry, the better the approximationworks.

9 The more skewness there is in the Distribution of the observations,the biggernneeds to the following plots, the histogram represents 10,000 simulated Xs, theblack curves are the true densities or CDFs, and the red curves are thenormal Limit Theorem21 Exponential samples n = samples n = samples n = samples n = samples n = samples n = Limit Theorem22 Gamma(5,1) samples n = (5,1) samples n = (5,1) samples n = (5,1) samples n = (5,1) samples n = (5,1) samples n = Limit samples n = 1XF(X) samples n = 2XF(X) samples n = 5XF(X) samples n = 10XF(X) samples n = 50XF(X) samples n = 100XF(X) Central Limit (5,1) samples n = 1XF(X) (5,1) samples n = 2XF(X) (5,1) samples n = 5XF(X) (5,1) samples n = 10XF(X) (5,1) samples n = 50XF(X) (5,1) samples n = 100XF(X) Central Limit Theorem25 There are other forms of the CLT, which relax the assumptions about thedistribution.

10 One example is, Theorem . [Liapunov s CLT]LetX1, X2, ..be independent randomvariables withE[Xi] = i,Var(Xi) = 2i, andE[|Xi |] = i. LetBn=(n i=1 i)1/3cn=(n i=1 2i)1/2= SD(n i=1Xi).ThenYn= ni=1(Xi i)cnD Z N(0,1)ifBncn condition involvingBiandcihas to do with each term in the sumhaving roughly the same weight. We don t want the sum to be dominatedby a few Limit Theorem26 Example: Regression through the originLetXi= weight of cariandYi= fuel in gallons to go 100 :Yi= Xi+ iwhere iare independent errors withE[ i] = 0,Var( i) = 2, E[| i|3]< 01000200030004000012345 WeightFuel UseCentral Limit Theorem27 How to estimate from data? WeightFuel UseMinimize the least squarescriterionSS( ) =n i=1(Yi Xi)2which is minimized by = ni=1 XiYi ni=1X2iWhat is the Distribution of ? = ni=1Xi( Xi+ i) ni=1X2i= + ni=1Xi i ni=1X2iCentral Limit Theorem28 LetZi=Xi i.