Overview - Department of Mathematics

1 NOTES ON PROBABILITYGreg LawlerLast Updated: March 21, 2016 OverviewThis is an introduction to the mathematical foundations of probability theory . It is intended as a supplementor follow-up to a graduate course in real analysis. The first two sections assume the knowledge of measurespaces, measurable functions, Lebesgue integral, and notions of convergence of functions; the third assumesFubini s Theorem; the fifth assumes knowledge of Fourier transform of nice (Schwartz) functions onR; andSection 6 uses the Radon-Nikodym mathematical foundations of probability theory are exactly the same as those of Lebesgue , probability adds much intuition and leads to different developments of the area. These notes areonly intended to be a brief introduction this might be considered what every graduate student shouldknow about the theory of uses some different terminology than that of Lebesgue integration inR. These notes willintroduce the terminology and will also relate these ideas to those that would be encountered in an ele-mentary (by which we will mean pre-measure theory ) course in probability or statistics.

2 Graduate studentsencountering probabilty for the first time might want to also read an undergraduate book in probability spacesDefinitionAprobability spaceis a measure space with total measure one. The standard notation is ( ,F,P)where: is a set (sometimes called asample spacein elementary probability ). Elements of are denoted and are sometimes calledoutcomes. Fis a -algebra (or -field, we will use these terms synonymously) of subsets of . Sets inFare calledevents. Pis a function fromFto [0,1] withP( ) = 1 and such that ifE1,E2,.. Fare disjoint,P j=1Ej = j=1P[Ej].We say probability ofE forP(E).Adiscrete probability spaceis a probability space such that is finite or countably infinite. In this casewe usually chooseFto be all the subsets of (this can be writtenF= 2 ), and the probability measurePis given by a functionp: [0,1] with p( ) = ,P(E) = Ep( ).We will consider another important example here, the probability space associated to an infinite numberof flips of a coin.

3 Let ={ = ( 1, 2,..) : j= 0 or 1}.1We think of 0 as tails and 1 as heads . For each positive integern, let n={( 1, 2,.., n) : j= 0 or 1}.Each nis a finite set with 2nelements. We can consider nas a probability space with -algebra 2 nandprobabilityPninduced bypn( ) = 2 n, the -algebra on consisting of all events that depend only on the firstnflips. More formally,we defineFnto be the collection of all subsetsAof such that there is anE 2 nwithA={( 1, 2,..) : ( 1, 2,.., n) E}.(1)Note thatFnis a finite -algebra (containing 22nsubsets) andF1 F2 F3 .IfAis of the form (1), we letP(A) =Pn(E).One can easily check that this definition is consistent. This gives a functionPonF0:= j= that analgebraof subsets of is a collection of subsets containing , closed under complementation,and closed underfiniteunions. To show closure under finite unions, it suffices to show that ifE1,E2 F0,thenE1 E2 an algebra but not a F0since F1.

4 SupposeE F0. ThenE Fnfor somen, and henceEc FnandEc , supposeE1,E2 F. Then there existsj,kwithE1 Fj,E2 Fk. Letn= max{j,k}. Then sincethe -algebras are increasing,E1,E2 Fnand henceE1 E2 Fn. Therefore,E1 E2 F0andF0is see thatF0is not a -algebra consider the singleton setE={(1,1,1,..)}.Eis not inF0butEcan be written as a countable intersection of events inF0,E= j=1{( 1, 2,..) : 1= 2= = j= 1}.Proposition functionPis a (countably additive) measure onF0, , it satisfiesP[ ] = 0,and ifE1,E2,.. F0are disjoint with n=1En F0, thenP[ n=1En]= n=1P(En). [ ] = 0 is immediate. Also it is easy to see thatPisfinitely additive, , ifE1,E2,..,En F0aredisjoint thenP n j=1Ej =n j=1P(Ej).(To see this, note that there must be anNsuch thatE1,..,En FNand then we can use the additivityofPN.)Showing countable subadditivity is harder. In fact, the following stronger fact holds: supposeE1,E2,.. F0are disjoint andE= n=1En F0. Then there is anNsuch thatEj= forj > N.

5 Once we establishthis, countable additivity follows from finite establish this, we consider as thetopologicalspace{0,1} {0,1} with the product topology where we have given each{0,1}the discrete topology (all four subsets are open).The product topology is the smallest topology such that all the sets inF0are open. Note also that all setsinF0are closed since they are complements of sets inF0. It follows from Tychonoff s Theorem that isa compact topological space under this topology. SupposeE1,E2,..are as above withE= n=1En ,E2,..is an open cover of the closed (and hence compact) setE. Therefore there is a finitesubcover,E1,E2,..,EN. Since theE1,E2,..are disjoint, this must imply thatEj= forj > the smallest -algebra containingF0. Then the Carath eodory Extension Theorem tells us thatPcan be extended uniquely to a complete measure space (P, F,P) whereF astute reader will note that the construction we just did is exactly the same as the construction ofLebesgue measure on [0,1].

6 Here we denote a real numberx [0,1] by its dyadic expansionx=. 1 2 3 = j=1 j2j.(There is a slight nuisance with the fact =.100000 , but this can be handled.) The -algebraFabove corresponds to the Borel subsets of [0,1] and the completionFcorresponds to theLebesgue measurable sets. If the most complicated probability space we were interested were the spaceabove, then we could just use Lebesgue measure on [0,1]. In fact, for almost all important applicationsof probability , onecouldchoose the measure space to be [0,1] with Lebesgue measure (see Exercise 3).However, this choice is not always the most convenient or on the last remark, one generally does not care what probability space one is working one observes are random variables which are discussed in the next Random Variables and ExpectationDefinitionArandom variableXis a measurable function from a probability space ( ,F,P) to the reals1, , it is a functionX: ( , )such that for every Borel setB,X 1(B) ={X B} we use the shorthand notation{X B}={ :X( ) B}.

7 IfXis a random variable, then for every Borel subsetBofR,X 1(B) F. We can define a functionon Borel sets by X(B) =P{X B}=P[X 1(B)].This function is in fact a measure, and (R,B, X) is a probability space. The measure Xis called thedistributionof the random variable. If Xgives measure one to a countable set of reals, thenXis calledadiscrete random variable. If Xgives zero measure to every singleton set, and hence to every countableset,Xis called acontinuous random variable. Every random variable can be written as a sum of a discreterandom variable and a continuous random variable. All random variables defined on a discrete probabilityspace are distribution Xis often given in terms of thedistribution function2defined byFX(x) =P{X x}= X( ,x].Note thatF=FXsatisfies the following: limx F(x) = 0. limx F(x) = 1. Fis a nondecreasing function. Fis right continuous, , for everyx,F(x+) := lim 0F(x+ ) =F(x).Conversely, anyFsatisfying the conditions above is the distribution function of a random variable.)

8 Thedistribution can be obtained from the distribution function by setting X( ,x] =FX(x),and extending uniquely to the Borel some continuous random variablesX, there is a functionf=fX:R [0, ) such thatP{a X b}= baf(x) a function, if it exists, is called thedensity3of the random variable. If the density exists, thenF(x) = x f(t) , more generally, to any topological space2often called cumulative distribution function (cdf) in elementary courses3 More precisely, it is the density or Radon-Nikoydm derivative with respect to Lebesgue measure. In elementary courses,the term probability density function (pdf) is often continuous att, then the fundamental theorem of calculus implies thatf(x) =F (x).A densityfsatisfies f(x)dx= , any nonnegative function that integrates to one is the density of a random an event, theindicator functionofEis the random variable1E( ) ={1, E,0, 6 E.(The corresponding function in analysis is often called the characteristic function and denoted E.)}

9 Proba-bilists never use the term characteristic function for the indicator function because the term characteristicfunction has another meaning. The term indicator function has no ambiguity.)ExampleLet ( ,F,P) be the probability space for infinite tossings of a coin as in the previous section. LetXn( 1, 2,..) = n={1,ifnth flip heads,0,ifnth flip tails.(2)Sn=X1+ +Xn= # heads on firstnflips.(3)ThenX1,X2,..,andS1,S2,..a re discrete random variables. IfFndenotes the -algebra of events thatdepend only on the firstnflips, thenSnis also a random variable on the probability space ( ,Fn,P).However,Sn+1is not a random variable on ( ,Fn,P).ExampleLet be any probability measure on (R,B). Consider the trivial random variableX=x,defined on the probability space (R,B, ). ThenXis a random variable and X= . Hence every probabilitymeasure onRis the distribution of a random random variableXhas anormal distributionwith mean and variance 2if it has densityf(x) =1 2 2e (x )2/2 2, < x <.}

10 If = 0 and 2= 1,Xis said to have astandard normal distribution. The distribution function of thestandard normal is often denoted , (x) = x 1 2 e t2 a random variable andg: (R,B) Ris a Borel measurable function, thenY=g(X) is also a random that the Cantor function is a continuous functionF: [0,1] [0,1] withF(0) = 0,F(1) = 1and such thatF (x) = 0 for allx [0,1]\AwhereAdenotes the middle thirds Cantor set. ExtendFtoRby settingF(x) = 0 forx 0 andF(x) = 1 forx 1. ThenFis a distribution function. A random variablewith this distribution function is continuous, sinceFis continuous. However, such a random variable has term random variable is a little misleading but it standard. It is perhaps easier to think of a randomvariable as a random number . For example, in the case of coin-flips we get an infinite sequence ofrandom numbers corresponding to the results of the a nonnegative random variable, theexpectationofX, denotedE(X), isE(X) = X dP,where the integral is the Lebesgue integral.