Lecture 6 Moment-generating functions

1 Lecture6: Moment-generating functions1of11 Course:Mathematical StatisticsTerm:Fall2017 Instructor:Gordan itkovi cLecture6 Moment-generating and first propertiesWe use many different functions to describe probability distribution (pdfs,pmfs, cdfs, quantile functions , survival functions , hazard functions , etc.) Moment-generating functions are just another way of describing distribu-tions, but they do require getting used as they lack the intuitive appeal ofpdfs or function (mgf)of the (dis-tribution of the) random variableYis the functionmYof a real param-etertdefined bymY(t) =E[etY],for allt Rfor which the expectationE[etY]is well is hard to give a direct intuition behind this definition, or to explain atwhy it is useful, at this point. It is related to the notions of Fourier transformand generating functions . It will be only through examples in this and laterlectures that a deeper understanding will first order of business is to compute the mgf for some of the more im-portant (named) random variables.

2 In the case of a continuous distribution,the main tool is the fundamental theorem which we use with the functiong(y) =exp(ty)- we think oftas fixed, so thatmY(t) =E[exp(tY)] =E[g(Y)] = g(y)fY(y)dy= etyfY(y) U(0, 1), so thatfY(y) =1{0 y 1}.Last Updated:September25,2019 Lecture6: Moment-generating functions2of11 ThenmY(t) = etyfY(y)dy= 10etydy=1t(et 1). us compute the mgf of the exponen-tial distributionY E( )with parameter >0:mY(t) = 0ety1 e y/ dy=1 0e y(1 t)dy=1 11 t=11 N(0, 1). As above,mY(t) = ety1 2 e integral looks hard to evaluate, but there is a simple trick. Wecollect the exponential terms and complete the square:etye 12y2=e 12(y t) we plug this into the expression above and pull oute12t2which isconstant, as far as the variable of integration is concerned, we getmY(t) =e12t2 1 2 e 12(y t) does not look like a big improvement at first, but it is.

3 Theexpression inside the integral is the pdf of a normal distributionwith meantand variance 1. Therefore, it must integrate to 1, asdoes any pdf. It follows thatmY(t) = you can see from the first part of this example, the moment generatingfunction does not have to be defined for allt. Indeed, the mfg of the expo-nential function is defined only fort<1 . We will not worry too much forabout this, and simply treat mgfs as expressions int, but this fact is good tokeep in mind when one goes deeper into the fundamental formula for continuous distributions becomes a sum inthe discrete case. WhenYis discrete with supportSYand pmfpY, the mgfcan be computed as follows, where, as above,g(y) =exp(ty):mY(t) =E[etY] =E[g(Y)] = y SYexp(ty)pY(y).Last Updated:September25,2019 Lecture6: Moment-generating functions3of11 Here are some Bernoulli B(p), we havemY(t) =et 0pY(0) +et 1pY(1) =q+pet,whereq=1 p.

4 Geometric g(p), thenP[Y=y] =qypandsomY(t) = y=0etypqy=p y=0(qet)y=p1 qet,where the last equality uses the familiar expression for the sumof a geometric series. We note that this only works forqet<1,so that, like the exponential distribution, the geometric distri-bution comes with a mgf defined only for some values oft. Poisson the Poisson distributionP( )with parameter >0, so thatSY={0, 1, 2, 3, ..}andpY(y) =e yy!. ThenmY(t) =E[exp(tY)] =E[g(Y)] = y SYg(y)pY(y)= y=0exp(ty)exp( ) yy!=e y=0(et )yy!The last sum on the right is nothing else by the Taylor formulafor the exponential function atx=et . Therefore,mY(t) =e (et 1).Here is how to compute the moment generating function of a linear trans-formation of a random variable. The formula follows from the simple factthatE[exp(t(aY+b))] =etbE[e(at)Y] that the random variable Y has the mgf mY(t).Then mgf of the random variable W=aY+b, where a and b are constants,is given bymW(t) =etbmY(at).

5 Last Updated:September25,2019 Lecture6: Moment-generating U(l,r). We can representWasW=aY+bwhereY U(0, 1),a= (r l)andb=l. We computedthe mgf ofYin -mY(t) =1t(et 1). Therefore,by (t) =etl1(r l)t(e(r l)t 1) =etr etat(r l). N( , ), thenWhas the same distri-bution as + Z, whereZ N(0, 1). Using the expression the mgf of aunitnormal distributionZ N(0, 1),we havemW(t) =e te12 2t2=e t+12 of independent random variablesOne of the most important properties of the Moment-generating functions isthat they turn sums of independent random variables into Y1,Y2, .. ,Ynbe independent random variables withmgfs mY1(t), mY2(t), .. , mYn(t). Then the mgf of their sum Y=Y1+Y2+ +Ynis given bymY(t) =mY1(t) mY2(t) mYn(t).This proposition is true for all random variables, but here is a sketch ofthe argument in the continuous case. It is a consequence of the factoriza-tion theorem (Theorem?)

6 ?) and the fundamental formula (Theorem??). Forsimplicity, let us assume thatn=2:mY1+Y2(t) =E[et(Y1+Y2)] =E[g(Y1,Y2)],whereg(y1,y2) =et(y1+y2). The factorization criterion says thatfY1,Y2(y1,y2) =Last Updated:September25,2019 Lecture6: Moment-generating functions5of11fY1(y1)fY2(y2), and, somY1+Y2(y) = et(y1+y2)fY1,Y2(y1,y2)dy2dy1= ety1ety2fY1(y1)fY2(y2)dy2dy1= ety1fY1(y1)( ety2fY2(y2)dy2)dy1= ety1fY1(y1)mY2(t)dy2=mY2(t) ety2fY2(y2)dy2=mY2(t)mY1(t). Binomial b(n,p). We know thatYcounts the number of successes innindependent Bernoulli trials, sowe can represent (in distribution) asY=Y1+ +Yn, where eachYiis aB(p)-random variable. We know from the mgfmYi(t)of eachYiisq+pet. ThereforemY(t) =mY1(t) mY2(t) mYn(t) = (q+pet) could have obtained the same formula without the factorizationcriterion, but the calculation is trickier:mY(t) =n y=0etypY(y) =n y=0ety(ny)pyqn y=n y=0(ny)(pet)yqn y= (pet+q)n,where the last inequality follows from thebinomial formula(a+b)n=n y=0(ny)aybn Moment-generating ?

7 The terminology moment generating function comes from the followingnice that the Moment-generating function mY(t)ofa random variable Y admits an expansion into a power series. Then theLast Updated:September25,2019 Lecture6: Moment-generating functions6of11coefficients are related to the moments of Y in the following way:mY(t) = k=0 kk!tk,( )where k=E[Yk]is the k-th moment of fully rigorous argument of this proposition is beyond the scope of thesenotes, but we can see why it works is we do the following formal computationbased on the Taylor expansion of the exponential functionex= k=0xkk!.We plug inx=tYand then take the expectation to getmY(t) =E[etY] =E[ k=0(tY)kk!] = k=0tk1k!E[Yk] = k=0 kk! ( ) suggests the following approach to the computation of mo-ments of a random variable:1. Compute the mgfmY(t).2. Expand it in a power series int, , writemY(t) = k= Set k=k!

8 Of the exponential know from the mgfmY(t)of the exponentialE( )-distribution is11 t. It is not hard to expand this into a power series because11 tis nothing by the sum of a geometric series11 t= k=0 follows immediately that k=k! Updated:September25,2019 Lecture6: Moment-generating of the uniform same example as abovetells us that the mgf of the uniform distributionU(l,r)ismY(t) =etr etlt(r l).We expand this into a series, buy expanding the numerator, first:etr etl= k=01k!(tr)k k=01k!(tl)k= k=0rk lkk! we divide by the denominatort(r l)to getmY(t) = k=0rk lkk!(r l)tk 1=1+r2 l22!(r l)t+r3 l33!(r l)t2+..It follows that k=rk+1 lk+1(k+1)(r l). a unit normal random variable, ,Y N(0, 1). We have computed its mgfet2/2above and we expandit using the Taylor formula for the exponential function:et2/2= k=01k!(t2/2)k= k=012kk! odd powers oftare all 0 so k=0 ifkis a moment of an even order 2k, we get 2k=(2k)!

9 2kk!.In all examples above we managed to expand the mfg into a power serieswithout using Taylor s theorem, , without derivatives. Sometimes, the eas-iest approach is to differentiate (the notation in ( ) means takekderiva-tives int, and then sett=0 ) k-th moment kof a random variable with themoment-generating function mY(t)is given by k=dkdtkmY(0)|t=0,( )as long as mYis defined for t in some neighborhood Updated:September25,2019 Lecture6: Moment-generating the Poisson random variable so thatmY(t) =e (et 1).The first derivativem Y(t)is given bye (et 1) etand, so 1=E[Y] =m Y(t)|t=0=e (e0 1) e0= .We can differentiate again to obtainm Y(t) = (1+et )et+ (et 1),which yields 2= (1+ ). One can continue and compute highermoments 3= (1+3 + 2), 4= (1+7 +6 2+ 3), is no simple formula for the general term the distributionIt is clear that different distributions come with different pdfs (pmf) and is also true for mgfs, but it is far from obvious and the proof is way outsidethe scope of these (Uniqueness theorem).

10 If two random variables Y1and Y2have the same moment generating functions , , ifmY1(t) =mY2(t)for all t,then the have the same distribution. In particular,1. if Y1is discrete, then so is Y2, and Y1and Y2have the same support andthe pmf, ,SY1=SY2and pY1(y) =pY2(y)for all If Y1is continuous, then so is Y2, and Y1and Y2have the same pdf, ,fY1(y) =fY2(y),for all way we use this result is straightforward:Last Updated:September25,2019 Lecture6: Moment-generating functions9of111. Find an expression for the mgf of the random variableYwhose distri-bution you don t know (most often, in this class, using a combination , but also see ).2. Look it up in a table of mgfs to identify course, for this to work you need to able to compute the mgf, and onceyou do, it has to be of the form you can recognize. When it works, it worksvery well, as the next example independent normal variables withmeans 1and 2, and standard deviations 1and 2.