Sample Spaces, Random Variables - Statistics

1 Sample Spaces, Random VariablesMoulinath BanerjeeUniversity of MichiganAugust 30, 20121 ProbabilitiesIn talking about probabilities, the fundamental object is , the Sample space . Points(elements) in are denoted (generically) by .We assign probabilities to subsets of .Assume for the moment that is finite or countably infinite. Thus could be the spaceof all possible outcomes when a coin is tossed three times in a row or say, the set of probabilityPis then a function from the power set (the class of all possible subsets)of , which we will denote byA, to the interval [0,1] satisfying the following properties: (i)P( ) = 1.

2 (ii)P( ) = 0. (ii) If{An}is a sequence of mutually disjoint subsets of , then,P( i=1Ai) = i=1P(Ai).In statistical/practical contexts it helps to think of as the set of all possible outcomesof a chance ( Random ) experiment. Probabilities assigned to subsets of are governed bythe nature of the chance mechanism (like tossing a coin, throwing a die, shuffling a pack ofcards etc.)1 Example:Consider a fair coin tossedntimes, where we recordHfor theith trialif the coin turns up Heads andTotherwise. The Sample space for this Random experimentis the set of all sequences ofH s andT s that have lengthn.

3 The size of is far as the assignment of probabilities is concerned, let denote a generic sequence ofHsandTs. If the outcomes of the various trials are independent of each otherP({ }) = 1 now the eventEthat the 1st toss yields anH. Since we are tossing a fair coinand the trials are independent, the probability of this is readily seen to be 1/2. Let s nowcompute the probability a different way. NowE= begins withH{ }and, therefore,P(E) = begins withHP({ }) =2n 12n= general however can be uncountably infinite (for example when is [0,1]) in whichcase (for certain technical reasons that we do not need to go into)Ais not taken to be theentire power set, but is chosen to be a smaller class.

4 Thus we do not assign probabilities toall subsets of , but only to those that belong toA. The classAis assumed to contain and to be closed under complementation and countable then, a function fromAto [0,1] with the same properties as above. The properties (i), (ii) and (iii) are called thebasic axiomsof these properties, it is easy to deduce the following: (a) For finitely many disjoint setsA1,A2,..,An,P( ni=1Ai) =n i=1P(Ai). (b)P(A) = 1 P(Ac). (c) IfA B, thenP(A) P(B). (d) For any two setsAandB(not necessarily disjoint),P(A B) =P(A) +P(B) P(A B).

5 Exercise: Using (i), (ii), (iii), deduce (a), (b), (c) and (d).There is an interesting generalization of (d) to the case of more than 2 :For (not necessarily disjoint) setsA1,A2,..An,P( Ai) = P(Ai) i<jP(Ai Aj) + i<j<kP(Ai Aj Ak) ..+ ( 1)n+1P( ni=1Ai).A general proof of this will be skipped. It can be done using induction. However, a proofof the above equality, when is a finite set and each element in is equally likely followsfrom a slick combinatorial , let ={1,2,..,N}and letPbe a probability such that for anyi,P(i) = 1 , clearly for any subsetA,P(A) = #(A)/N.

6 Proving the above proposition then boilsdown to establishing that,#( Ai) = #(Ai) i<j#(Ai Aj) + i<j<k#(Ai Aj Ak) ..+ ( 1)n+1#( ni=1Ai).So consider some elementsbelonging to Ai. We need to show thatsis counted exactlyonce on the right side of the above expression. Suppose thatsbelongs tokof , on the right side of the above expressionsis countedk (k2)+(k3) ..+ ( 1)k+1(kk)times. Call this numberm. Then,m=k j=1( 1)j+1(kj)= 1 1 +k j=1( 1)j+1(kj)= 1 (1 k j=1( 1)j(kj))= 1 (k j=0(kj)( 1)j(1)k j)= 1 (1 1)k= finishes the probability: Consider a fixed eventBwithP(B)>0.

7 The conditionalprobability ofA|B(read,AgivenB) is,P(A|B) =P(A B)P(B).It is not difficult to show that this defines a valid probability. In other words if we define afunctionPBfromAto [0,1] as follows:PB(A) =P(A|B),thenPBsatisfies the basic axioms of key idea behind defining conditional probabilities is to update the probabilities ofevents given the knowledge thatBhappened. Note that for any eventC Bc,P(C|B) = 0,which is in accordance with common sense. On the other hand, for anyC B,P(C|B) P(C) ;in other words, the conditional probability of a sub-event ofB, given thatBhappened isgenerally higher than the unconditional probabilities can be expressed in terms of conditional and marginal probabilities inthe following manner:P(A B) =P(A|B)P(B),providedP(B)>0.

8 Also,P(A B) =P(B|A)P(A),providedP(A)>0. More generally, for setsA1,A2,..An, we have,P( ni=1Ai) =P(A1)P(A2|A1)P(A3|A1 A2).. P(An| n 1i=1Ai).One of the most important consequences of the notion of conditional probability isBayes Rule. Simple as it looks, its ramifications are profound. Bayes Rule gives us away of obtaining the conditional probability of one event given another, when we know themarginal probabilities and the conditional probability of the second event given the state Bayes Rule in the form of the following :Suppose that{Bj}is a (finite/infinite) partition of andAis any event4withP(A)>0.

9 Also suppose thatP(Bi)>0 for eachi(without any loss of generality).Then,P(Bj|A) =P(Bj A)P(A)=P(A|Bj)P(Bj) P(A|Bi)P(Bi).Illustrating Bayes Rule:There are three cabinets, A, B and C, each of which hastwo drawers. Each drawer has one coin. A has two gold coins, B has two silver coins and Chas one gold and one silver coin. A cabinet is chosen at Random ; one drawer is opened anda silver coin found. What is the chance that the other drawer also has a silver coin ?To fit this in the framework of the above theorem, take theBj s to be the events thatAis chosen,Bis chosen andCis chosen.

10 For brevity we shall denote the events byA,BandC. Note that these events are indeed a partition of the sure event that a cabinet is chosen,and furthermoreP(A) =P(B) =P(C) = 1 denote the event that the opened drawer of the chosen cabinet has a silver , we are required to findP(B|S1). Now, using Bayes Rule, we haveP(B|S1) =P(S1|B)P(B)P(S1|A)P(A) +P(S1|B)P(B) +P(S1|C)P(C)=1 1/30 1/3 + 1 1/3 + 1/2 1/3=11 + 1/2= 2 of events:EventsAandBare said to be independent ifP(A B) =P(A) P(B). In general, eventsA1,A2.