Lecture 3 : Probability Theory - MIT OpenCourseWare

1 Lecture 3 : Probability and reviewWe consider real-valued discrete random variables and continuous ran-dom variables. A discrete random variableXis given by itsprobabilitymass function which is a non-negative real valued functionfX: R 0satisfyingx fX(x) = 1 for some finite domain known as thesample space. For example,2/3 (x=1)fX(x) = 1/3 (x= 1),0(otherwise)denotes the Probability mass function of a discrete random variableXwhichtakes value 1 with Probability 2/ 3 and 1 with Probability 1 continuous random variableYis given by itsprobability density func-tionwhich is a non-negative real valued functionfY: R 0satisfying fY(y)dy=1 (we will mostly consider cases when the sample space is the realsR). For example,1 (y[0,1])fY(y) ={ ,0 (otherwise)denotes the Probability density function of a continuous random variableYwhich takes a uniform value in the interval [0,1].}

2 For a given setA , one can compute the Probability of the eventsX AandY AasP(X A) = fX(x),P(Y A) = A fY(y) (or mean) of a random variableis defined asE[X]= xfX(x),E[Y] =x yfY(y) distribution functionof a random variable is defined asFX(x) =P(X x).Two random variablesXandYareindependentifP({X A} {Y B}) =P(X A)P(Y B)12for all setsAandB. Random variablesX1,X2, ,Xnaremutually inde-pendentifnP({X1 A1} {X2 A2} {Xn An}) = P(Xii=1 Ai),for allsetsAi. Theyarepairwise independentifXiandXjare independentfor alli6=j. Two random variables areuncorrelatedifE[XY] =E[X]E[Y]. and lognormal first review one of themost important distributions in Probability reals < < and >0, the normal distribution(or Gaussian distribution) denotedN( , 2), with mean and variance 2is a continuous random variable with Probability density function1f(x) = 2e (x )/(2 2)(< x <).

3 2 Exercise , by computation, that the mean of the normal distri-bution is .To modelthe stock market, it is more reasonable to assert that the rateof change of the stock price has normal distribution (compared to the stockprice itself having normal distribution). Log-normal distribution can be usedto model such situation, where a log-normal distribution is a probabilitydistribution whose (natural) logarithm has normal distribution. In otherwords, if the cumulative distribution function of normal distribution isF(x),then that for log-normaldistribution isF(lnx).To obtain the probablility distribution of the log-normal distribution, wecan use the change of variable (Change of variable) Suppose thatX( ) is a randomvariable over reals with Probability distributionfX(x)( (y))andcumulative distribution functionFX(x)( (y)).

4 Further suppose thatFX,FYare differentiable, and there exists a functionhsuch thatFY(y) =FX(h(y)). ThendFYfY(y) =dy=dFXLecture 3=fX(h(y)) h (y).dyThe termh (y)is called the Jacobian. (to avoid unnecessary technicality,we assume thatall functions are differentiable)We can now compute the Probability density function of log-normal dis-tribution (or define it using this formula).Definition reals < < and >0, the log-normal distri-bution with parameters and is a distribution with Probability density3functiong(x) =1x 2e (lnx )/(2 2), x > Exercise (i) Compute theexpectation and variance of log-normal dis-tribution (HINT: it suffices to compute the case when = 0. You can use thefact thatf(x) given above is a in order to simply the computation).(ii) Showthat the product of two independent log-normal distributions isalso a log-normal examples of other importatnt distributions that will repeatedly oc-cur throughout the course are Poisson distribution and exponential distribu-tion.

5 All these distributions (including the normal and log-normal distribu-tion) are examples ofexponential familieswhich are defined as distributionswhose Probability distribution function can be written in terms of a vectorparameter as follows:f( )(x) =h(x)c( )exp(for kwi( )ti(x)i=1),somec( ) 0 andreal-valued functionsw1( ), ,wk( ). For example,the pdfof log-normal distribution can be written as1g(x) =x 1 2 exp( (lnx)22 2+ (lnx) 2 22 2),and thus for = ( , ), we may leth(x) =12,c( ) =1 2 e 2/2 2,w1( ) = 12 2,t1(x) = (lnx)2,w2( ) = Lecture 32, andt2(x) = lnx. Exponential families are known toexhibit many nice statistical this course, we will mostly be interested in the statistics of a random variable. Here are two topics that we will address today.(1) Moment generating function : thek-th moment of a random vari-able is defined asEXk.

6 The moments of a random variable containessential statiscial information about the random variable. The mo-ment generating function is a way of encoding these information intoa single function.(2) Long-term (large-scale) behavior : we will study the outcome ofrepeated independent trials (realizations) of a same random generating functionDefinition (Moment generating function) A moment generating func-tion of a given random variableXis a functionMX:R RdefinedasMX(t) =E[etX].We note thatnot all random varaibles allow a moment generating func-tion, since the sum on the right hand side might not converge. For example,4the log-normal distribution does not have a moment generating function. Infact,E[etX] does not exist for allt6= name moment generating function comes from the fact that thek-thmoment can be compted asEXk=dkMX(0)dxkfor allk 0.

7 Therefore, themoment generating function (if exists) can be also written as MX(t) = tkk=0mk,k!wheremk=EXkis thek-th moment (i) LetXandYbe two random variable with the samemoment generation function, ,MX(t) =MY(t)for allt. ThenXandYhavethe same distribution, , the cumulative distribution functions arethe same.(ii) Suppose thatXis a random variable with moment generating func-tionMX(t)and continuous cumulative distribution functionFX(x). LetX1,X2, ,be a sequence of random variables such thatlimMXi(t) =MX(t),i for allt. ThenXiconverges toXin distribution, , for all realx, we havelimFXi(x) =FX(x).i We willnot prove this theorem here. See [1] for its should be careful when considering part (i) of this theorem. It doesnot imply that all two random variables with same moments have the samedistribution (see [1] page 106-107).

8 This can happen because the momentgenerating function need not of large numbersTheorem (Weak law of large number) Suppose that (indepen-dent identically distributed) random variablesX1,X2,21 ,Xnof mean andvariance are given, and letX=n(X1+ +Xn). Then for all positive ,P(|X | ) 0asn . linearity of expectation, we see thatE[X] =1nn i=1E[Xi] = ,and thusE[X ] = 0. Next step is to compute the variance ofX. We haveV[X] =E[(X )2] =E[(1 Lecture 3 n(Xini=1 ))2],5which by the fact thatXiare independent, becomesE[(X )2] =1n2E[n i=1(Xi )2]= 2P(|X | ) E[(X )2] = 2n,we see thatP(|X | ) V[X] 2= 2,n 2where the right hand side tends to zero asntends to infinity. The theorem stated above is in a very weak form;(1) the conditions can be relaxed and still give the same conclusion,(2) a stronger conclusion can be deduced from the same assumption(strong law of large number).

9 See [1] for more the games played in the casino that you play againstthe casino (for example, roulette) are designed so that casino has about1 5% advantage over the player when the player plays the optimal are also designed so that the variance is large compared to thisaverage gain. Hence from the player s perspective, it looks as if anythingcan happen over a small period of time and that one has a reasonable chanceof making money. From the casino s perspective, as long as they can provideindependent outcomes for each game, the law of large number tells that theyare going to make money. This does not apply to games that player playswith each other ( , poker). For these games, the casino makes money bycharging certain amount of fee for playing each limit theoremIn the law of large numbers, we considered the random variableYn=1n ni=1Xi, which has varianceV[Yn] = 2n.

10 Suppose that the mean ofeachXiis zero so thatYnhas mean 0, and consider the random variableZn= nYn. Note thatZnhas mean 0 and variance 2, which is the same asXi. Will the random variableZnbehave similarly toXi? If not, can we sayanything interesting about it? The following theorem provides a surprisinganswer to this (Central limit theorem) LetX1,X2, be randomvariables with mean and variance 2and moment generating functionM(t). Then asntends to infinity, the random variableZn= n((1 Lecture 3nn i=1Xi) )converges in distribution to the normal distributionN(0, 2). may replaceXiwithXi , and assume that = 0. Considerthemoment generating function ofZn:1 MZn(t) =E[etZn=Et][e in nXi=1].SinceXiare independent random variables, we havenMZn(t) = E[etXi/i=1n] =M(t n) (t tk) =nk=0nk/2k!EXk= 1 + 2t22n+O(1n3/2) =e 2t2/(2n)+O(1/n3/2),we haveM(t 2n)n=(e t2/(2n)+O(1/n3/2)n2=e t2/2+O(1/n1/2).)