Summary of basic probability theory Math 218, …

1 Summary of basic probability theoryMath 218, mathematical StatisticsD Joyce, Spring 2016 Sample spaceconsists of aun-derlyingset , whose elements are calledoutcomes,a collection of subsets of calledevents, and afunctionPon the set of events, called aprobabilityfunction, satisfying the following The probability of any event is a number inthe interval [0,1].2. The entire set is an event with probabilityP( ) = The union and intersection of any finite orcountably infinite set of events are events, and thecomplement of an event is an The probability of a disjoint union of a finiteor countably infinite set of events is the sum of theprobabilities of those events,P( iEi) = iP(Ei).From these axioms a number of other propertiescan be derived including The complementEc= Eof an eventEisan event, andP(Ec) = 1 P(E).

2 6. The empty set is an event with probabilityP( ) = For any two eventsEandF,P(E F) =P(E) +P(F) P(E F),thereforeP(E F) P(E) +P(F).8. For any two eventsEandF,P(E) =P(E F) +P(E Fc).9. If eventEis a subset of eventF, thenP(E) P(F).10. Statement 7 above is called theprinciple ofinclusion and exclusion. It generalizes to more thantwo (n r=1Er)=n i=1P(Ei) i<jP(Ei Ej)+ i<j<kP(Ei Ej Ek) + ( 1)n 1P(E1 E2 En)In words, to find the probability of a union ofnevents, first sum their individual probabilities,then subtract the sum of the probabilities of alltheir pairwise intersections, then add back the sumof the probabilities of all their 3-way interections,then subtract the 4-way intersections, and continueadding and subtractingk-way intersections untilyou finally stop with the probability of variables order to de-scribe a sample space, we frequently introduce asymbolXcalled arandom variablefor the sam-ple this notation, we can replacethe probability of an event,P(E)

3 , by the notationP(X E), which, by itself, doesn t do much. Butmany events are built from the set operations ofcomplement, union, and intersection, and with therandom variable notation, we can replace those bylogical operations for not , or , and and . For in-stance, the probabilityP(E Fc) can be writtenasP(X EbutX / F).Also, probabilities of finite events can be writ-ten in terms of equality. For instance, the prob-ability of a singleton,P({a}), can be written asP(X=a), and that for a doubleton,P({a,b}) =P(X=aorX=b).One of the main purposes of the random variablenotation is when we have two uses for the same1sample space. For instance, if you have a fair die,the sample space is ={1,2,3,4,5,6}where theprobability of any singleton is16.

4 If you have twofair dice, you can use two random variables,XandY, to refer to the two dice, but each has the samesample space. (Soon, we ll look at the joint distri-bution of (X,Y), which has a sample space definedon .Random variables and cumulative distribu-tion sample space can have any setas its underlying set, but usually they re relatedto numbers. Often the sample space is the set ofreal numbersR, and sometimes a power of the most common sample space only has two el-ements, that is, there are only two outcomes. Forinstance, flipping a coin as two outcomes Headsand Tails; many experiments have two outcomes Success and Failure; and polls often have twooutcomes For and Against. Even though theseevents aren t numbers, it s useful to replace themby numbers, namely 0 and 1, so that Heads, Suc-cess, and For are identified with 1, and Tails, Fail-ure, and Against are identified with 0.)

5 Then thesample space can haveRas its underlying the sample space does haveRas its un-derlying set, the random variableXis called arealrandom variable. With it, the probability of an in-terval like [a,b], which isP([a,b]), can then be de-scribed asP(a X b). Unions of intervals canalso be described, for instanceP(( ,3) [4,5])can be written asP(X <3 or 4 X 5).When the sample space isR, the probabilityfunctionPis determined by a cumulative distri-bution function ( )Fas follows. The functionF:R Ris defined byF(x) =P(X x) =P(( ,x]).Then, fromF, the probability of a half-open inter-val can be found asP((a,b]) =F(b) F(a).Also, the probability of a singleton{b}can be foundas a limitP({b}) = lima b(F(b) F(a)).From these, probabilities of unions of intervals canbe computed.))

6 Sometimes, the is simply calledthedistribution, and the sample space is identifiedwith this sample distribu-tions are determined entirely by the probabilities oftheir outcomes, that is, the probability of an eventEisP(E) = x EP(X=x) = x EP({x}).The sum here, of course, is either a finite or count-ably infinite sum. Such a distribution is called adis-crete distribution, and when there are only finitelymany outcomesxwith nonzero probabilities, it iscalled afinite discrete distributions is usually described interms of a probability mass function ( )fde-fined byf(x) =P(X=x) =P({x}).This is enough to determine this distributionsince, by the definition of a discrete distribution,the probability of an eventEisP(E) = x Ef(x).In many applications, a finite distribution isuni-form, that is, the probabilities of its outcomes areall the same, 1/n, wherenis the number of out-comes with nonzero probabilities.

7 When that isthe case, the field of combinatorics is useful in find-ing probabilities of events. Combinatorics includesvarious principles of counting such as the multipli-cation principle, permutations, and the cumula-tive distribution functionFfor a distribution isdifferentiable function, we say it s acontinuous dis-tribution. Such a distribution is determined by aprobability density functionf. The relation be-tweenFandfis thatfis the derivativeF ofF,andFis the integral (x) = x f(t)dtConditional probability and two events, withP(F)6= 0, thentheconditional probabilityofEgivenFis definedto beP(E|F) =P(E F)P(F).Two events,EandF, neither with probability0, are said to beindependent, ormutually indepen-dent, if any of the following three logically equiva-lent conditions holdsP(E F) =P(E)P(F)P(E|F) =P(E)P(F|E) =P(F)Bayes formula is useful to invertconditional probabilities.

8 It saysP(F|E) =P(E|F)P(F)P(E)=P(E|F)P(F)P(E|F)P(F) +P(E|Fc)P(Fc)where the second form is often more useful in valueE(X), alsocalled theexpectationormean X, of a randomvariableXis defined differently for the discrete andcontinuous a discrete random variable, it is a weightedaverage defined in terms of the probability massfunctionfasE(X) = X= xxf(x).For a continuous random variable, it is defined interms of the probability density functionfasE(X) = X= xf(x) is a physical interpretation where thismean is interpreted as a center of is a linear operator. That meansthat the expectation of a sum or difference is thedifference of the expectationsE(X+Y) =E(X) +E(Y),and that s true whether or notXandYare inde-pendent, and alsoE(cX) =cE(X)wherecis any constant.

9 From these two propertiesit follows thatE(X Y) =E(X) E(Y),and, more generally, expectation preserves linearcombinationsE(n i=1ciXi)=n i=1ciE(Xi).Furthermore, whenXandYare independent,thenE(XY) =E(X)E(Y), but that equationdoesn t usually hold whenXandYare not and standard a random variableXis defined asVar(X) = 2X=E((X X)2) =E(X2) 2 Xwhere the last equality is provable. Standard devia-tion, , is defined as the square root of the are a couple of properties of variance. First,if you multiply a random variableXby a constantcto getcX, the variance changes by a factor of thesquare ofc, that isVar(cX) =c2 Var(X).3 That s the main reason why we take the squareroot of variance to normalize it the standard de-viation ofcXisctimes the standard deviation ofX.

10 Also, variance is translation invariant, that is,if you add a constant to a random variable, thevariance doesn t change:Var(X+c) = Var(X).In general, the variance of the sum of two randomvariables isnotthe sum of the variances of the tworandom variables. But it is when the two randomvariables are , central moments, skewness, a random variableXis defined as k=E(Xk). Thus, the mean isthe first moment, = 1, and the variance canbe found from the first and second moments, 2= 2 momentis defined asE((X ) , the variance is the second central third central moment of the standardized ran-dom variableX =X , 3=E((X )3) =E((X )3) 3is called theskewnessofX. A distribution that ssymmetric about its mean has 0 skewness. (In factall the odd central moments are 0 for a symmetricdistribution.))