A Tutorial on Probability Theory

1 A Tutorial on Probability TheoryPaola SebastianiDepartment of Mathematics and StatisticsUniversity of Massachusetts at AmherstCorresponding Author:Paola Sebastiani. Department of Mathematics and statistics , Universityof Massachusetts, Amherst, MA 01003, :(413)545 0622,fax:(413) 545 Tutorial on Probability TheoryContents1 Probability and Uncertainty22 Basic Definitions23 Basic Axioms34 Conditional Probability55 Bayes Theorem66 Independence and Conditional Independence77 Discrete Random Variables88 Continuous Random Variables129 Multivariate Distributions1510 Summaries1911 Special Distributions2312 Independence23 References231A Tutorial on Probability Theory1.

2 Probability and UncertaintyProbability measures the amount of uncertainty of an event: a fact whose occurrence is , as an example, the eventR Tomorrow, January 16th, it will rain in Amherst . Theoccurrence ofRis difficult to predict we have all been victims of wrong forecasts made by the weather channel and we quantify this uncertainty with a numberp(R), called the probabilityofR. It is common to assume that this number is non-negative and it cannot exceed 1. The twoextremes are interpreted as the Probability of the impossible event:p(R) = 0, and the probabilityof the sure event:p(R) = 1.

3 Thus,p(R) = 0 asserts that the eventRwill not occur while, on theother hand,p(R) = 1 asserts thatRwill occur with now that you are asked to quote the Probability ofR, and your answer isp(R) = are two main interpretations of this number. The ratio represent the odds in favor ofR. This is thesubjective probabilitythat measures your personal belief probabilityis the interpretation ofp(R) = as a relative frequency. Suppose, for instance, that in the lastten years, it rained 7 times on the day 16th January.

4 Then = 7/10 is the relative frequency ofoccurrences ofR, also given by the ratio between the favorable cases (7) and all possible cases (10).There are other interpretations ofp(R) = arising, for instance, from logic or psychology (seeGood (1968) for an overview.) Here, we will simply focus attention to rules for computations Basic DefinitionsDefinition 1 (Sample Space)The set of all possible events is called thesample spaceand isdenoted we denote events by capital lettersA, B, .., we writeS={A, B.}

5 }. The identification of thesample space depends on the problem at hand. For instance, in the exercise of forecasting tomorrowweather, the sample space consists of all meteorological situations: rain (R), sun (S), cloud (C),typhoon (T) sample space is a set, on which we define some algebraic operations between 2 (Algebraic Operations)LetAandBbe two events of the sample spaceS. We willdenote Adoes not occur by A; eitherAorBoccur byA B; bothAandBoccur byA, B; Aoccurs andBdoes not byA\B A, eventsAandBareexhaustiveifA B=S, in other words we are sure that eitherAorBwilloccur.

6 Thus, in particularA A=S. The eventsAandBareexclusiveifA, B= , where is theimpossible event, that is the event whose occurrence is known to be impossible. In this case, we aresure that ifAoccurs thenBcannot. Clearly, we haveA, A= .2A Tutorial on Probability TheoryA, BA 1: Graphical representation of operations with operations with events are easily represented via Venn s diagrams. Conventionally, we willrepresent events as rectangles, whose area is their Probability . BecauseSis the union of all possibleevents, its Probability isp(S) = 1 and we representSas a square of side 1.

7 Under this conventionalrepresentation, we will call Athe complementation ofA, that is A=S\A. Similarly, the eventsA B,A, BandA\Bwill be called the union, the intersection and the difference 1 (Venn s Diagrams)Figure 1 gives an example of a Venn s diagram. The rectangleof height and of area represents the eventA, withp(A) = Therectangle of height and the eventB, withp(B) = The event Ais the rectangle with , and Bis given by union of the rectangles with The intersection ofAandBis given by the rectangle with , so thatp(A, B) = The unionA Bis given by the rectangle with (A B) = eventA\Bis represented by the rectangle with , andB\Aby the rectangle Thus, we havep(A\B) = (B\A)

8 = Basic AxiomsIn Example 1 the Probability of an event is the area of the rectangle that represents the event, andthe sample space is the union of all events. This representation can be generalized to more abstract3A Tutorial on Probability Theoryspaces and leads to an axiomatic definition of Probability (Kolmogorov, 1950) in terms of measureover a collection of subsets. This collection is assumed to contain the empty set, and to be closedunder the complementation and countable union ( i=1Ai S.)Theorem 1 LetSdenote the sample space.

9 A set functionp( )defined inSis a Probability any eventAinS, thenp(A) 0; (S) = 1; , A2, ..are exclusive events inSand henceAi, Aj= for alli, j, thenp( i=1Ai) = i=1p(Ai).From these axioms, the following elementary properties can be derived. See Karr (1992, page 25)for 1 Letp( )be a Probability function defined over the sample spaceS. Thenp( )satisfiesthe following ( ) = 0; ( )isfinitely additive: ifA1, A2, .. , Anare events inS, such thatAi, Aj= for alli6=j,thenp( nh=1Ah) =n h=1p(Ah).(1)If these events form a partition ofS, they are such that nh=1Ah=S, thenp( nh=1Ah) = 1; (A A) =p(A) +p( A) = 1, so thatp(A) 1for anyAinS; Bthenp(B\A) =p(B) p(A);Axiom(iii)is known ascountable additivityand it is rejected by a school of probabilists who replacethe countable additivity by finite additivity.

10 Further details are in DeFinetti (1972).Consider now the two eventsAandBin Figure 1. If we computedp(A B) asp(A) +p(B)we would obtainp(A B) = that exceeds 1. The error here is that, in computingp(A B) asp(A) +p(B), the eventA, Bis counted twice. Indeed, we can decomposeAinto (A\B) (A, B) andsimilarlyBinto (A, B) (B\A). Since the intersection (A\B),(A, B) = , the events (A\B) and(A, B) are exclusive and there follows, from item 3 in Theorem 1, thatp(A) =p(A\B)+p(A, B), andsimilarlyp(B) =p(A, B) +p(B\A).