Notes on Probability Theory and Statistics

1 Notes on Probability Theory andStatisticsAntonis Demos(Athens University of Economics and Business)October 20022 Part IProbability Theory3 Chapter Set Theory DigressionAsetis defined as any collection of objects, which are biggest possible collection of points under consideration is called thespace,universe,oruniversal set. For Probability Theory the space is called called asubsetofB(we writeA BorB A) if every elementofAis also an element called aproper subsetofB(we writeA BorB A) if every element ofAis also an element ofBand there is at least one elementofBwhich does not belong setsAandBare calledequivalent setsorequal sets(we writeA=B)ifA BandB a set has no points, it will be called theemptyornullset and denoted by .Thecomplementof a setAwith respect to the space , denoted by A,Ac,or A, is the set of all points that are in but not two setsAandBis a set that consists of the commonelements of the two sets and it is denoted byA two setsAandBis a set that consists of all points that are inAorBor both (but only once) and it is denoted byA differenceof two setsAandBis a set that consists of all points in6 IntroductionAthat are not inBand it is denoted byA of Set OperationsCommutative:A B=B AandA B=B :A (B C)=(A B) CandA (B C)=(A B) :A (B C)=(A B) (A C)andA (B C)=(A B) (A C).

2 (Ac)c= A = the complement of theA-complement of (the space) then:A =A,A = ,A = ,A =A,A A= ,A A= ,A A=A,andA A= Morgan Law:(A B)=A B,and(A B)=A or mutually exclusivesets are the sets that their intersection isthe empty set, mutually exclusive ifA B= . SubsetsA1,A2, ..are mutually exclusive ifAi Aj= for anyi6= or variability are prevalent in many situations and it is the purposeof the Probability Theory to understand and quantify this notion. The basic situationis an experiment whose outcome is unknown before it takes place , a) coin tossing,b)throwingadie,c)choosingatrando manumberfromN,d)choosingatrandomanumber from(0,1).Thesample spaceis the collection or totality of all possible outcomes of aconceptual experiment. Aneventis a subset of the sample space. The class of allevents associated with a given experiment is defined to be theevent us describe the sample spaceS, the set of all possible relevant outcomesof the above experiments, ,S={H,T},S={1,2,3,4,5,6}.

3 In both of theseexamples we have afinite sample space. In example c) the sample space is a countableinfinity whereas in d) it is an uncountable or a priori Probability : If a random experiment can result inNmutually exclusive and equally likely outcomes and ifN(A)of these outcomes have anattributeA,thentheprobabilityofAis the fractionN(A) (A)=N(A)/N,Set Theory Digression7whereN=N(A)+N(A).Example:Cons ider the drawing an ace (eventA) from a deck of 52 isP(A)?We have thatN(A)=4andN(A)= (A)+N(A)=4+48=52andP(A)=N(A)N=452 Frequency or a posteriori Probability : Is the ratio of the number thatan eventAhas occurred out ofntrials, (A)= :Assume that weflip a coin 1000 times and we observe 450 the a posteriori Probability isP(A)= /n=450/1000 = (this is also therelative frequency). Notice that the a priori Probability is in this case Probability : This is based on intuition or shall be concerned with a priori probabilities.

4 These probabilities involve,many times, the counting of possible Some Counting ProblemsSome more sophisticated discrete problems require counting techniques. For example:a) What is the Probability of getting four of a kind in afive card poker?b) What is the Probability that two people in a classroom have the samebirthday?The sample space in both cases, although discrete, can be quite large and itnot feasible to write out all possible Duplication is permissible and Order is important (MultipleChoice Arrangement), the elementAAis permitted andABis a differentelement fromBA. In this case where we want to arrangenobjects inxplaces thepossibleoutcomesisgivenfrom:Mnx= :Find all possible combinations of the letters A, B, C, and D whenduplication is allowed and order is :n=4,andx=2, consequently the8 Introductionpossible number of combinations isM42=42= the result we can also Duplication is not permissible and Order is important (Per-mutation Arrangement), the elementAAisnotpermitted andABis adifferent element fromBA.

5 In this case where we want to permutenobjects inxplaces the possible outcomes is given from:Pnxor P(n, x)=n (n 1) ..(n x+1)=n!(n x)!.Example:Find all possible permutations of the letters A, B, C, and D whenduplication is not allowed and order is :n=4,andx=2, consequently thepossible number of combinations isP42=4!(4 2)!=2 3 42= Duplication is not permissible and Order is not important(Combination Arrangement), the elementAAisnotpermitted andABisnotadifferent element fromBA. In this case where we want the combinations ofnobjects inxplaces the possible outcomes is given from:Cnxor C(n, x)=P(n, x)x!=n!(n x)!x!= nx Example:Find all possible combinations of the letters A, B, C, and D whenduplication is not allowed and order is not :n=4,andx=2, consequently thepossible number of combinations isC42=4!

6 2! (4 2)!=2 3 42 2= us now define Probability Definition of ProbabilityConsider a collection of setsA with index , which is denoted by{A : }.We can define for an index of arbitrary cardinality (the cardinal number of a set isthe number of elements of this set): A ={x S:x A for some }Set Theory Digression9 A ={x S:x A for all }A collection is exhaustive if A =S(partition), and is pairwise exclusiveor disjoint ifA A = , 6= .To define probabilities we need some further structure. This is because inuncountable cases we can not just define Probability for all subsets ofS,asthereare some sets on the real line whose Probability can not be determined, , they areunmeasurable. We shall define Probability on a family of subsets ofS,ofwhichwerequire the following 1 Let beAa non-empty class of subsets an algebra A, wheneverA A2 A,wheneverA1,A2 a -algebra if also2/.

7 N=1An A, wheneverAn A, n=1,2,3,..Note that sinceAis non-empty, (1) and (2) AandS n=1An -algebra is the set of all subsets ofS, denoted byP(S), and the smallest is{ , S}.Wecangeneratea -algebra from any collectionof subsets by adding to the set the complements and the unions of its elements. Forexample letS=R,andB={[a, b],(a, b],[a, b),(a, b),a,b R},and letA= (B)consists of all intervals and countable unions of intervals andcomplements thereof. This is called the Borel -algebra and is the usual -algebrawe work whenS= -algebraA P(R), , there are sets inP(R)notinA. These are some pretty nasty ones like the Cantor set. We can alternativelyconstruct the Borel -algebra by consideringJthe set of all intervals of the form( ,x],x R. We can prove that (J)= (B). Wecannowgivethedefinitionof Probability measure which is due to 2 Given a sample spaceSand a -algebra(S,A), a Probability measureis a mapping fromA Rsuch (A) 0for allA (S)=13.)

8 IfA1,A2, ..are pairwise disjoint, ,Ai Aj= for alli6=j,thenP [i=1Ai!= Xi=1P(Ai)Insuchawaywehaveaprobabilityspa ce(S,A,P).WhenSis discrete weusually takeA=P(S).WhenS=Ror some subinterval thereof, we takeA= (B).Pis a matter of choice and will depend on the problem. In many discretecases, the problem can usually be written such that outcomes are equally ({x})=1/n, n=`(S).In continuous cases,Pis usually like Lebesgue measure, ,P((a, b)) b ( )= (A) (Ac)=1 P(A) (B Ac)=P(B) P(B A)5. IfA B P(A) P(B) (B A)=+P(A)+P(B) P(A B)More generally, for eventsA1,A2, ..An Awe have:P"n[i=1Ai#=nXi=1P[Ai] XXi<jP[AiAj]+XXXi<j<kP[AiAjAk] ..+( 1)n+1P[ ].Forn=3the above formula is:PhA1[A2[A3i=P[A1]+P[A2]+P[A3] P[A1A2] P[A1A3] P[A2A3]+P[A1A2A2].Conditional Probability and ( i=1Ai) P i=1P(Ai)Proofs involve manipulating sets to obtain disjoint sets and then apply Conditional Probability and IndependenceIn many statistical applications we have variablesXandY(or eventsAandB)and want to explain or predictYorAfromXorB, we are interested not only inmarginal probabilities but in conditional ones as well, , we want to incorporatesome information in our predictions.]]]]

9 LetAandBbe two events inAand a probabilityfunctionP(.).Theconditional probabilityofAgiven eventB,isdenotedbyP[A|B]and is defined as follows:Definition 3 The Probability of an eventAgivenaneventB, denoted byP(A|B),is given byP([A|B)=P(A B)P(B)if P(B)>0and is left undefined ifP(B)= the above formula is evidentP[AB]=P[A|B]P[B]=P[B|A]P[A]ifboth P[A]andP[B]are nonzero. Notice that when speaking of conditional proba-bilities we are conditioning on some given eventB; that is, we are assuming that theexperiment has resulted in some outcome ,ineffect then becomes our new sample space. All Probability properties of the previous section apply to conditionalprobabilities as well, ( |B)is a Probability measure. In (A|B) (S|B)= ( i=1Ai|B)=P i=1P(Ai|B)for any pairwise disjoint events{Ai} i= that ifAandBare mutually exclusive events,P(A|B)= B, P(A|B)=P(A)P(B) P(A)with strict inequality unlessP(B)= A, P(A|B)= , there is an additional property (Law) called theLaw of TotalProbabilitieswhich states that:LAW OF TOTAL Probability :P(A)=P(A B)+P(A Bc)For a given Probability space( ,A,P[.)]]

10 ]),ifB1,B2, .., Bnis a collection of mutuallyexclusive events inAsatisfyingnSi=1Bi= andP[Bi]>0fori=1,2, .., nthen foreveryA A,P[A]=nXi=1P[A|Bi]P[Bi]Another important theorem in Probability is the so calledBayes Theoremwhich states:BAYES RULE: Given a Probability space( ,A,P[.]),ifB1,B2, .., Bnis acollection of mutually exclusive events inAsatisfyingnSi=1Bi= andP[Bi]>0fori=1,2,..,nthen for everyA Afor whichP[A]>0we have:P[Bj|A]=P[A|Bj]P[Bj]nPi=1P[A|Bi]P[B i]Notice that for eventsAandB Awhich satisfyP[A]>0andP[B]>0we have:P(B|A)=P(A|B)P(B)P(A|B)P(B)+P(A|Bc) P(Bc).This follows from the definition of conditional independence and the law of totalprobability. The probabilityP(B)is a prior Probability andP(A|B)frequently is alikelihood, whileP(B|A)is the theMultiplication Rulestates:Given a Probability space( ,A,P[.