10 Moment generating functions - Mathematics Home

1 10 Moment generating FUNCTIONS11910 Moment generating functionsIfXis a random variable, then itsmoment generating functionis (t) = X(t) =E(etX) ={ xetxP(X=x) in discrete case, etxfX(x)dxin continuous thatXis Exponential(1) random variable, that is,fX(x) ={e xx >0,0x , (t) = 0etxe xdx=11 t,only whent <1. Otherwise the integral diverges and the Moment generating function does notexist. Have in mind that Moment generating function is only meaningful when the integral (orthe sum) is where the name comes from. Writing its Taylor expansion in place ofetXandexchanging the sum and the integral (which can be done in manycases)E(etX) =E[1 +tX+12t2X2+13!]}}

2 T3X3+..]= 1 +tE(X) +12t2E(X2) +13!t3E(X3) +..The expectation of thek-th power ofX,mk=E(Xk), is called thek-thmomentofx. Incombinatorial language, then, (t) is the exponential generating function of the also thatddtE(etX)|t=0=EX,d2dt2E(etX)|t=0=EX2 ,which lets you compute the expectation and variance of a random variable once you know itsmoment generating the Moment generating function for a Poisson( ) random Moment generating FUNCTIONS120By definition, (t) = n=0etn nn!e =e n=0(et )nn!=e + et=e (et 1).Example the Moment generating function for a standard Normal definition, X(t) =1 2 etxe x2/2dx=1 2 e12t2 e 12(x t)2dx=e12t2,where from the first to the second line we have used, in the exponent,tx 12x2= 12( 2tx+x2) =12((x t)2 t2).

3 Lemma ,X2,..,Xnare independent andSn=X1+..+Xn, then Sn(t) = X1(t).. Xn(t).IfXiis identically distributed asX, then Sn(t) = ( X(t)) follows from multiplicativity of expectation for independent random variables:E[etSn] =E[etX1 etX2 .. etXn] =E[etX1] E[etX2] .. E[etXn].Example the Moment generating functions of a Binomial(n,p) random we haveSn= nk=1 IkwhereIkare independent andIk=I{success onkth trial}, so that Sn(t) = (etp+ 1 p) Moment generating FUNCTIONS121 Why are Moment generating functions useful? One reason is the computation oflarge devia-tions. LetSn=X1+ +Xn, whereXiare independent and identically distributed asX, withexpectationEX= and Moment generating function.

4 At issue is the probability thatSnisfar away from its expectationn , more preciselyP(Sn> an), wherea > . We can of courseuse Chebyshev s inequality to get a bound of order1n. But it turns out that this probabilitytends to be much deviation that (t)is finite for somet >0. For anya > ,P(Sn an) exp( nI(a)),whereI(a) = sup{at log (t) :t >0}> anyt >0, using the Markov s inequality,P(Sn an) =P(etSn tan 1) E[etSn tan] =e tan (t)n= exp ( n(at log (t))).Note thatt >0 is arbitrary, so we can optimize overtto get what the theorem claims. We needto show that (a)>0 whena > . For this, note that (t) =at log (t) satisfies (0) = 0and, assuming that one can differentiate under the integral sign (which one can in this case butproving this requires a bit of abstract analysis beyond our scope), (t) =a (t) (t)=a E(XetX) (t),and then (0) =a >0,so that (t)>0 for some small enough a fair dientimes and letSnbe the sum of the numbers you roll.

5 Estimatethe probability thatSnexceeds its expectation by at leastn, forn= 100 andn= fit this into the above theorem: observe that = and soESn= , and that weneed to find an upper bound onP(Sn ), ,a= Moreover (t) =166 i=1eit=et(e6t 1)6(et 1).and we need to computeI( ), which by definition is the maximum, overt >0, of the log (t),whose graph is in the figure Moment generating would be nice if we could solve this problem by calculus, but unfortunately we cannot (whichis very common in such problems), so we resort to numerical calculations. The maximum is att and as a resultI( ) is a little larger than This gives the upper boundP(Sn ) e n,which is about forn= 10, 10 8forn= 100, and 10 78forn= 1000.

6 The bound3512nfor the same probability, obtained by Chebyshev s inequality, is much much too large reason why Moment generating functions are useful is that they characterize thedistribution, and convergence of distributions. We will state the following theorem that the Moment generating functions for random variablesX,Y, andXnare finite for If X(t) = Y(t)for allt, thenP(X x) =P(Y x)for If Xn(t) X(t)for allt, andP(X x)is continuous inx, thenP(Xn x) P(X x)for that the sum of independent Poisson random variables is the situation, then. We havenindependent random variablesX1,..,Xn, such that:X1is Poisson( 1), X1(t) =e 1(et 1),X2is Poisson( 2), X2(t) =e 2(et 1).

7 Xnis Poisson( n), Xn(t) =e n(et 1).10 Moment generating FUNCTIONS123 Then X1+..+Xn(t) =e( 1+..+ n)(et 1)and soX1+..+Xnis Poisson( 1+..+ n). Very similarly, one could also prove that thesum of independent Normal random variables is will now reformulate and prove the Central Limit Theorem in a special case when momentgenerating function is finite. This assumption is not needed, and you should apply it as we didin the previous thatXis a random variable withEX= andVar(X) = 2, andassume that X(t)is finite for allt. LetSn=X1+..+Xn, whereX1,..,Xnare i. i. d., anddistrubuted asX. LetTn=Sn n , for everyx,P(Tn x) P(Z x),asn , whereZis a standard Normal random andYi=Xi.

8 ThenYiare independent, distributed asY,E(Yi) = 0,Var(Yi) = 1, andTn=Y1+..+Yn finish the proof, we show that Tn(t) Z(t) = exp(t2/2) asn : Tn(t) =E[et Tn]=E[et nY1+..+t nYn]=E[et nY1] E[et nYn]=E[et nY]n=(1 +t nEY+12t2nE(Y2) +16t3n3/2E(Y3) +..)n=(1 + 0 +12t2n+16t3n3/2E(Y3) +..)n (1 +t221n)n Moment generating FUNCTIONS124 Problems1. The player pulls three cards at random from a full deck, andcollects as many dollars as thenumber of red cards among the three. Assume 10 people each play this game once, and letXbe the number of their combined winnings. Compute the momentgenerating function Compute the Moment generating function of a uniform random variable on [0,1].

9 3. This exercise was in fact the original motivation for the study of large deviations, by theSwedish probabilist Harald Cram`er, who was working as an insurance company consultant in1930 s. Assume that the insurance company receives a steadystream of payments, amountingto (a deterministic number) per day. Also every day, they receive a certain amount in claims;assume this amount is Normal with expectation and variance 2. Assume also day-to-dayindependence of the claims. The regulators require that within a period ofndays, the companymust be able to cover its claims by the payments received in the same period,or else.

10 Intimidatedby the fierce regulators, the company wants to fail to satisfythe regulators with probability lessthan some small number . The parametersn, , and are fixed, but is something thecompany controls. Determine .4. Assume thatSnis Binomial(n,p). For everya > p, determine by calculus the large deviationbound forP(Sn an).5. Using the central limit theorem for a sum of Poisson randomvariables, computelimn e nn i=0nii!.Solutions to problems1. Compute the Moment generating function for a single game,then raise it to the 10th power: (t) =(1(523)((263)+(261)(262) et+(262)(261) e2t+(263) e3t)) Answer: (t) = 10etxdx=1t(et 1).