LECTURE NOTES on PROBABILITY and STATISTICS Eusebius …

1 LECTURE NOTESonPROBABILITY and STATISTICSE usebius DoedelTABLE OF CONTENTSSAMPLE SPACES1 Events5 The Algebra of Events6 Axioms of Probability9 Further Properties10 Counting Outcomes13 Permutations14 Combinations21 CONDITIONAL PROBABILITY45 Independent Events63 DISCRETE RANDOM VARIABLES71 Joint distributions82 Independent random variables91 Conditional distributions97 Expectation101 Variance and Standard Deviation108 Covariance110 SPECIAL DISCRETE RANDOM VARIABLES118 The Bernoulli Random Variable118 The Binomial Random Variable120 The Poisson Random Variable130 CONTINUOUS RANDOM VARIABLES142 Joint distributions150 Marginal density functions153 Independent continuous random variables158 Conditional distributions161 Expectation163 Variance169 Covariance175 Markov s inequality181 Chebyshev s inequality184 SPECIAL CONTINUOUS RANDOM VARIABLES187 The Uniform Random Variable187 The Exponential Random Variable191 The Standard Normal Random Variable196 The General Normal Random Variable201 The Chi-Square Random Variable206 THE CENTRAL LIMIT THEOREM211 SAMPLE STATISTICS246 The Sample Mean252 The Sample Variance257 Estimating the Variance of a Normal Distribution266 Samples from Finite Populations274 The Sample Correlation Coefficient282 Maximum Likelihood Estimators288 Hypothesis Testing305 LEAST SQUARES APPROXIMATION335 Linear Least Squares335 General Least Squares343 RANDOM NUMBER GENERATION362 The Logistic Equation363 Generating Random Numbers378 Generating Uniformly Distributed Random Numbers379 Generating Random Numbers using the Inverse Method392 SUMMARY TABLES AND FORMULAS403 SAMPLE SPACESDEFINITION:Thesample spaceis the set of all possible outcomes of an.

2 When weflip a cointhen sample space isS={H , T},whereHdenotes that the coin lands Heads up andTdenotes that the coin lands Tails up .For a fair coin we expect H and T to have the same chance ofoccurring, , if we flip the coin many times then about 50 % of theoutcomes will say that theprobabilityof H to occur is (or 50 %) .The PROBABILITY ofTto occur is then also :When weroll a fair diethen the sample space isS={1,2,3,4,5,6}.The PROBABILITY the die lands withkup is16, (k= 1,2, ,6).When we roll it 1200 times we expect a 5 up about 200 PROBABILITY the die lands with aneven numberup is16+16+16= :When we toss a coin 3 times and record the results in thesequencethat they occur, then the sample space isS={HHH , HHT , HTH , HTT , THH , THT , TTH , TTT}.Elements ofSare vectors , sequences , or ordered outcomes .We may expect each of the 8 outcomes to be equally the PROBABILITY of the PROBABILITY of a sequence to contain precisely two Headsis18+18+18= : When we toss a coin 3 times and record the resultswithout paying attention to the order in which they occur, , if weonly record the number of Heads, then the sample space isS=n{H,H,H},{H,H,T},{H,T,T},{T,T,T} outcomes inSare nowsets; , order is not that the ordered outcomes are{HHH , HHT , HTH , HTT , THH , THT , TTH , TTT}.

3 Note that{H,H,H}corresponds tooneof the ordered outcomes,{H,H,T},,three,,{H,T,T},,three, ,{T,T,T},,one,,Thus{H,H,H}and{T,T,T}each occur with probability18,while{H,H,T}and{H,T,T}each occur with PROBABILITY Theory subsets of the sample space are : The set of basic outcomes of rolling a dieonceisS={1,2,3,4,5,6},so the subsetE={2,4,6}is an example of an a die is rolledonceand it lands with a 2ora 4ora 6 up then wesay that the have already seen that the PROBABILITY thatEoccurs isP(E) =16+16+16= Algebra of EventsSince events aresets, namely, subsets of the sample spaceS, we cando the usualset operations:IfEandFare events then we can formEcthecomplementofEE FtheunionofEandFEFtheintersectionofEandF We writeE FifEis : In PROBABILITY Theory we useEcinstead of E,EFinstead ofE F,E Finstead ofE the sample spaceSisfinitethen we typically allow any subset ofSto be an : If we randomly drawone characterfrom a box con-taining the charactersa,b, andc, then the sample space isS={a , b , c},and there are 8 possible events, namely, those in the set of eventsE=n{ },{a},{b},{c},{a,b},{a,c},{b,c},{a,b,c} the outcomesa,b, andc, are equally likely to occur, thenP({ }) = 0, P({a}) =13, P({b}) =13, P({c}) =13,P({a,b}) =23, P({a,c}) =23, P({b,c}) =23, P({a,b,c}) = example,P({a,b}) is the PROBABILITY the character is always assume that the setEof allowable eventsincludes thecomplements, unions, and intersectionsof its : If the sample space isS={a , b , c , d},and we start with the eventsE0=n{a},{c,d}o,then this set of events needs to be extended to (at least)E=n{},{a},{c,d},{b,c,d},{a,b},{a,c ,d},{b},{a,b,c,d}.

4 VerifyEincludes complements, unions, of ProbabilityAprobability functionPassigns a real number (theprobabilityofE)to every eventEin a sample ( ) must satisfy the following basic properties : 0 P(E) 1 , P(S) = 1 , For anydisjoint eventsEi, i= 1,2, ,n,we haveP(E1 E2 En) =P(E1) +P(E2) + P(En).9 Further PropertiesPROPERTY 1:P(E Ec) =P(E) +P(Ec) = 1.(Why ?)ThusP(Ec) = 1 P(E).EXAMPLE:What is the PROBABILITY of at least one H infour tossesof a coin?SOLUTION: The sample spaceSwill have 16 outcomes. (Which?)P(at least one H) = 1 P(no H) = 1 116= 2:P(E F) =P(E) +P(F) P(EF).PROOF(using the third axiom) :P(E F) =P(EF) +P(EFc) +P(EcF)= [P(EF) +P(EFc)] + [P(EF) +P(EcF)] P(EF)= P(E) + P(F) - P(EF) .(Why ?)NOTE: Draw a Venn diagram withEandFto see this ! The formula is similar to the one for the number of elements :n(E F) =n(E) +n(F) n(EF).11So far our sample spacesShave also becountably infinite, , the setZof all also beuncountable, , the setRof all real : Record the low temperature in Montreal on January8 in each of a large number of can takeSto be the set ofall real numbers, ,S=R.

5 (Are there are other choices ofS?)What PROBABILITY would you expect for the followingeventsto have?(a)P({ })(b)P({x: < x < })(How does this differ from finite sample spaces?)We will encounter such infinite sample spaces many times 12 Counting OutcomesWe have seen examples where the outcomes in afinitesample spaceSareequally likely, , they havethe same sample spaces occur quite probabilities then requires countingalloutcomes andcountingcertain typesof outcomes .The counting has to be done carefully!We will discuss a number of representative examples in that arise Here we count of the number of words that can be formedfrom a collection of items ( , letters). (Also calledsequences,vectors,ordered sets .) Theorderof the items in the word is important; , the wordacbis different from the wordbac. Theword lengthis the number of characters in the :Forsetsthe order is not important. For example, the set{a,c,b}isthe same as the set{b,a,c}.

6 14 EXAMPLE: Suppose that four-letter words oflower casealpha-betic characters are generated randomly with equally likely outcomes.(Assume thatletters may appear repeatedly.)(a) How many four-letter words are there in the sample spaceS?SOLUTION: 264= 456,976 .(b) How many four-letter words are there are there inSthat startwith the letter s ?SOLUTION: 263.(c) What is theprobabilityof generating a four-letter word thatstarts with an s ?SOLUTION:263264=126 = this have been computed more easily?15 EXAMPLE: How many re-orderings (permutations) are there ofthe stringabc? (Hereletters may appear only once.)SOLUTION: Six, namely,abc,acb,bac,bca,cab, these permutations are generated randomly with equal probabilitythen what is the PROBABILITY the word starts with the letter a ?SOLUTION:26= : In general, if the word length isnandall charactersare distinctthen there aren! permutations of the word.

7 (Why ?)If these permutations are generated randomly with equal probabilitythen what is the PROBABILITY the word starts with a particular letter ?SOLUTION:(n 1)!n!=1n.(Why ?)16 EXAMPLE: How manywordsof lengthkcan be formed froma set ofn(distinct) characters ,(wherek n) ,when letters can be usedat most once?SOLUTION:n(n 1) (n 2) (n (k 1))=n(n 1) (n 2) (n k+ 1)=n!(n k)!(Why ?)17 EXAMPLE:Three-letter wordsare generated randomly from thefivecharactersa , b , c , d , e, where letters can beused at mostonce.(a) How many three-letter words are there in the sample spaceS?SOLUTION: 5 4 3 = 60 .(b) How many words containinga , bare there inS?SOLUTION: First place the charactersa , ,select the two indicesof the locations to place can be done in3 2 = 6 ways.(Why ?)There remains one position to be filled with ac , dor the number of words is 3 6 = (c) Suppose the 60 solutions in the sample space areequally is theprobabilityof generating a three-letter word thatcontains the lettersaandb?

8 SOLUTION:1860= :Suppose the sample spaceSconsists of allfive-letterwordshavingdistinct alphabetic characters. How many words are there inS? How many special words are inSfor whichonlythe secondand the fourth character are vowels, , one of{a,e,i,o,u,y}? Assuming the outcomes inSto be equally likely, what is theprobability of drawing such a special word?20 CombinationsLetSbe a set containingn(distinct) fromS,isany selectionofkelements fromS,whereorder is not important.(Thus the selection is aset.)NOTE: By definition asetalways hasdistinct :There are threecombinationsof 2 elements chosen from the sets ={a , b , c},namely, thesubsets{a,b},{a,c},{b,c},whereas there are sixwordsof 2 elements fromS,namely,ab , ba , ac , ca , bc , cb .22In general, givena setSofnelements ,the number of possible subsets ofkelements fromSequals nk n!k! (n k)!.REMARK: The notation nk is referred to as nchoosek .NOTE: nn =n!

9 N! (n n)!=n!n! 0!= 1,since 0! 1 (by convenient definition !) .23 PROOF:First recall that there aren(n 1) (n 2) (n k+ 1) =n!(n k)!possiblesequencesofkdistinct elements , every sequence of lengthkhask! permutations of itself,and each of these defines the same subset the total number of subsets isn!k! (n k)! nk .24 EXAMPLE:In the previous example, with 2 elements chosen from the set{a , b , c},we haven= 3 andk= 2 , so that there are3!(3 2)!= 6words,namelyab , ba , ac , ca , bc , cb ,while there are 32 3!2! (3 2)!=62= 3subsets,namely{a,b},{a,c},{b,c}.25 EXAMPLE: If we choose 3 elements from{a , b , c , d},thenn= 4 andk= 3,so there are4!(4 3)!= 24 words,namely :abc , abd , acd , bcd ,acb , adb , adc , bdc ,bac , bad , cad , cbd ,bca , bda , cda , cdb ,cab , dab , dac , dbc ,cba , dba , dca , dcb ,while there are 43 4!3! (4 3)!=246= 4 subsets,namely,{a,b,c},{a,b,d},{a,c,d},{ b,c,d}.

10 26 EXAMPLE:(a) How many ways are there to choose a committee of 4 personsfrom a group of 10 persons, if order is not important?SOLUTION: 104 =10!4! (10 4)!= 210.(b) If each of these 210 outcomes is equally likely then what is theprobability that a particular person is on the committee?SOLUTION: 93 / 104 =84210=410.(Why ?)Is this result surprising?27(c) What is the PROBABILITY that a particular person isnoton thecommittee?SOLUTION: 94 / 104 =126210=610.(Why ?)Is this result surprising?(d) How many ways are there to choose a committee of 4 personsfrom a group of 10 persons, if one is to be the chairperson?SOLUTION: 101 93 = 10 93 = 109!3! (9 3)!= : Why is this four times the number in (a) ?28 EXAMPLE:Two ballsare selected at random from a bag withfour whiteballs andthree blackballs, where order is not would be an appropriate sample spaceS?SOLUTION: Denote the set of balls byB={w1, w2, w3, w4, b1, b2, b3},where same color balls are made distinct by numbering a good choice of the sample space isS= the set ofall subsetsoftwo ballsfromB ,because the wording selected at random suggests that each suchsubset has the same chance to be number of outcomes inS(which are sets of two balls) is then 72 = : (continued )(Two ballsare selected at random from a bag withfour whiteballsandthree blackballs.)