Probability Theory - Bard College

1 Probability TheoryProbability Spaces and EventsConsider a random experiment with several possible outcomes. For example, wemight roll a pair of dice, flip a coin three times, or choose a random real numberbetween 0 and 1. Thesample spacefor such an experiment is the set of all possibleoutcomes. For example: The sample space for a pair of die rolls is the set{1,2,3,4,5,6} {1,2,3,4,5,6}. The sample space for three coin flips is the set{0,1}3, where 0 represents headsand 1 represents tails. The sample space for a random number between 0 and 1 is the interval [0,1].Aneventis any statement about the outcome of an experiment to which we canassign a Probability . For example, if we roll a pair of dice, possible events include: Both dice show even numbers. The first die shows a 5 or 6. The sum of the values shown by the dice is greater than or equal to 7. From a formal point of view, events are usually defined to be certain subsets of thesample space.

2 Thus the event both dice show even numbers refers to the subset{2,4,6} {2,4,6}. Despite this, it is more common to write statements than subsetswhen referring to a specific the special case of an experiment with finitely many outcomes, we can definethe Probability of any subset of the sample space, and therefore every subset is anevent. In the general case, however, Probability is a measure on the sample space,and only measurable subsets of the sample space are following definition serves as the foundation of modern Probability Theory :Definition: Probability SpaceAprobability spaceis an ordered triple ( ,E,P), where: is a set (thesample space). Elements of are calledoutcomes. Eis a -algebra over . Elements ofEare calledevents. P:E [0,1] is a measure satisfyingP( ) = 1. This is theprobabilitymeasureon .In general, a measure :M [0, ] on a setSis called aprobability measureif (S) = 1, in which case the triple(S,M, )forms a Probability 1 Finite Probability SpacesConsider an experiment with finitely many outcomes 1.

3 , n, with correspondingprobabilitiesp1,..,pn. Such an experiment corresponds to a finite Probability space( ,E,P), where: is the set{ 1,.., n}, Eis the collection of all subsets of , and P:E [0,1] is the Probability measure on defined by the formulaP({ i1,.., ik}) =pi1+ +pik. EXAMPLE 2 The Interval[0,1]Consider an experiment whose outcome is a random number between 0 and 1. Wecan model such an experiment by the Probability space ( ,E,P), where: is the interval [0,1], Eis the collection of all Lebesgue measurable subsets of [0,1], and P:E [0,1] is Lebesgue measure on [0,1].Using this model, the Probability that the outcome lies in a given setE [0,1] isequal to the Lebesgue measure ofE. For example, the Probability that the outcome isrational is 0, and the Probability that the outcome lies between and is 1/10. EXAMPLE 3 Product SpacesLet ( 1,E1,P1) and ( 2,E2,P2) be Probability spaces corresponding to two possible3(a)(b)Figure 1: (a) Each infinite sequence of coin flips corresponds to a path down aninfinite binary tree.

4 In this case, the sequence begins with 010. (b) The leaves of aninfinite binary tree form a Cantor Now, imagine that we perform both experiments, recording the outcomefor each. The combined outcome for this experiment is an ordered pair ( 1, 2), where 1 1and 2 2. In fact, the combined experiment corresponds to a probabilityspace ( ,E,P), where: is the Cartesian product 1 2. Eis the -algebra generated by all events of the formE1 E2, whereE1 E1andE2 E2. P:E [0,1] is the product of the measuresP1andP2. That is,Pis the uniquemeasure with domainEsatisfyingP(E1 E2) =P1(E1)P2(E2) for allE1 E1andE2 example, if we pick two random numbers between 0 and 1, the correspondingsample space is the square [0,1] [0,1], with the Probability measure being two-dimensional Lebesgue measure. EXAMPLE 4 The Cantor SetSuppose we flip an infinite sequence of coins, recording 0 for heads and 1 for outcome of this experiment will be an infinite sequence (1,0,1,1,0,1,0,0,1.)

5 Of 0 s and 1 s, so the sample space is the infinite product{0,1} the sample space{0,1}Nis infinite-dimensional, we can visualize eachoutcome as a path down an infinite binary tree, as shown in Figure 1a. If we imaginethat this tree has leaves at infinity, then each outcome corresponds to one suchleaf. Indeed, it is possible to draw this tree so that the leaves are visible, as shownin Figure 1b. As you can see, the leaves of the tree form a Cantor set. Indeed, under4the product topology, the sample space ={0,1}Nis homeomorphic to the standardmiddle-thirds Cantor is not too hard to put a measure on . Given a finite sequenceb1,..,bnof 0 sand 1 s, letB(b1,..,bn) be the set of outcomes whose firstnflips areb1,..,bn, anddefineP0(B(b1,..,bn))= the collection of all such sets, and letP (E) = inf{ P0(Bn) B1,B2,.. BandE Bn}for everyE . ThenP is an outer measure on , and the resulting measurePisa Probability measure.

6 The mechanism described above for putting a measure on{0,1}Ncan be modifiedto put a measure on Nfor any Probability space ( ,E,P). For example, it is possibleto talk about an experiment in which we roll an infinite sequence of dice, or pick aninfinite sequence of random numbers between 0 and 1, and for each of these there isa corresponding Probability VariablesArandom variableis a quantity whose value is determined by the results of arandom experiment. For example, if we roll two dice, then the sum of the values ofthe dice is a random general, a random variable may take values from any : Random VariableArandom variableon a Probability space ( ,E,P) is a functionX: S,whereSis any the case whereS=R(or more generally ifSis a topological space), weusually require a random variable to be ameasurablefunction, 1(U) should bemeasurable for every open setU 5 Here are some basic examples of random variables: If we roll two dice, then the valuesXandYthat show on the dice are randomvariables {1,2,3,4,5,6}.

7 Expressions involvingXandY, such as the sumX+Yor the quantityX2+Y3, are also random If we pick a random number between 0 and 1, then the numberXthat we pickis a random variable [0,1]. For an infinite number of coin flips, we can define a sequenceC1,C2,..ofrandom variables {0,1}byCn={0 if thenth flip is heads,1 if thenth flip is tails. Although a random variable is defined as a function, we usually think of it as avariable that depends on the outcome . In particular, we will often writeXwhen we really meanX( ). For example, ifXis a real-valued random variable, thenP(X 3)would refer toP({ |X( ) 3}).Definition: Distribution of a Random VariableThedistributionof a random variableX: Sis the Probability measurePXonSdefined byPX(T) =P(X T).for any measurable setT expressionP(X T) in the definition above refers to the Probability thatthe value ofXis an element ofT, the Probability of the eventX 1(T). Thus thedistribution ofXis defined by the equationPX(T) =P(X 1(T))Note that the setX 1(T) is automatically measurable sinceXis a measurable func-tion.}

8 In measure-theoretic terms,PXis the pushforward of the Probability measurePby to the 6 Die RollLetX: {1,2,3,4,5,6}be the value of a die roll. Then for any subsetTof{1,2,3,4,5,6}, we havePX(T) =P(X T) =|T| is,PXis the Probability measure on{1,2,3,4,5,6}where each point has prob-ability 1/6. 6 EXAMPLE 7 Random Real NumberLetX: [0,1] be a random real number between 0 and 1. AssumingXis equallylikely to lie in any portion of the interval [0,1], the distributionPXis just Lebesguemeasure on [0,1]. This is known as theuniform distributionon [0,1]. The most basic type of random variable is a discrete variable:Definition: Discrete Random VariableA random variableX: Sis said to bediscreteifSis finite or : Sis discrete, then the Probability distributionPXforXis completelydetermined by the Probability of each element ofS. In particular:PX(T) = x TPX({x})for any 8 Difference of DiceLetXandYbe random die rolls, and letZ=|X Y|.

9 ThenZis a random variable {0,1,2,3,4,5}, with the following Probability distribution:PZ({0}) =16, PZ({1}) =1036, PZ({2}) =836,PZ({3}) =636, PZ({4}) =436, PZ({5}) =236. We end with a useful formula for integrating with respect to a Probability distri-bution. This is essentially just a restatement of the formula for the Lebesgue integralwith respect to a pushforward 1 Integration Formula for DistributionsLetX: Sbe a random variable, and letg:S Rbe a measurable Sg dPX= g(X)dP,whereg(X)denotes the random variableg X: Random VariablesOne particularly important type of random variable is a continuous variable : Continuous Random VariableLetX: Rbe a random variable with Probability distributionPX. We say thatXiscontinuousif there exists a measurable functionfX:R [0, ] such thatPX(T) = TfXdmfor every measurable setT R, wheremdenotes Lebesgue measure. In this case,the functionfXis called aprobability density is,Xis continuous ifPXis absolutely continuous with respect to Lebesguemeasure, ifdPX=fXdmfor some non-negative measurable functionfX.

10 Recall the following formula for inte-grals with respect to such 2 Weighted IntegrationLetX: Rbe a continuous random variable with Probability densityfX, andletg:R Rbe a measurable function. Then Rg dPX= 9 Standard Normal DistributionConsider a random variableX: Rwith Probability density function defined byfX(x) =1 2 exp( 12x2).In this case,Xis said to have thestandard normal distribution. A graph of thefunctionfXis shown in Figure such anX, the probabilityPX(T) that the valueXlies in any setTis givenby the formulaPX(T) = (a)PHa<X<bLab(b)Figure 2: (a) The Probability densityfXfor a standard normal distribution. (b) Areaunder the graph offXon the interval (a,b).For example, if (a,b) is an open interval, thenP(a < X < b) = (a,b)fXdm= ba1 2 exp( 12x2) is illustrated in Figure 2b. The Probability density functionfXcan be thought of as describing the probabilityper unit length ofPX. The following theorem gives us a formula for this function:Theorem 3 Density FormulaLetX: Rbe a random variable, and define a functionfX:R [0, ]byfX(x) = limh 0+PX([x h,x+h]) (x)is defined for almost allx RandfXis measurable.