MATH 2P82 MATHEMATICAL STATISTICS (Lecture Notes)

1 MATH 2P82 MATHEMATICAL STATISTICS (Lecture Notes) c Jan Vrbik23 Contents1 PROBABILITY REVIEW7 Basic 7 Binomial 7 Multinomial 7 Random Experiments (Basic Definitions).. 7 Sample 8 Set 8 Boolean 8 Probability of 8 Probability 9 Important 9 Probability 9 Product 9 Conditional 9 Total-probability 10 Discrete Random 10 Bivariate (joint) 11 Conditional 11 Multivariate 11 Expected Value of a 11 Expected values related 12 Moments (univariate).. 12 Moments (bivariate or joint ).. 12 Variance ofaX+bY+ 13 Moment generating 13 Main 13 Probability generating 13 Conditional expected 14 Common discrete 14 Negative 15 Multivariate 15 Continuous Random 16 Univariate probability density function (pdf).. 16 Distribution 16 Bivariate (multivariate) 16 Marginal 16 Conditional 17 Mutual 17 Expected 17 Common Continuous 18 Transforming Random 192 Transforming Random Variables21 Univariate 21 Distribution-Function(F) 21 Probability-Density-Function(f) 23 Bivariate 24 Distribution-Function 24 Pdf (Shortcut) 253 Random Sampling31 Sample 31 Central Limit 31 Sample 32 Sampling fromN( , ).

2 33 Sampling without 35 Bivariate 364 OrderStatistics37 Univariate 37 Sample 38 Bivariate 40 Special 415 Estimating Distribution 45 Cram r-Rao 50 Method of 51 One 52 Two 53 Maximum-likelihood 53 One 5556 Confidence Intervals57CI for mean .. 57 58 Large-sample 58 Difference of two 58 Proportion(s).. 59 Variance(s).. 60 607 Testing Hypotheses61 Tests concerning mean(s).. 62 Concerning variance(s).. 63 Concerning proportion(s).. 63 Contingency 63 Goodness 638 Linear Regression and Correlation65 Simple 65 Maximum likelihood 65 Least-squares 65 Normal 66 Statistical properties of the 67 Confidence 70 Multiple 71 Various standard 739 AnalysisofVariance75 One-way 75 Two-way 76No 77 With 7810 Nonparametric Tests79 Sign 79 Signed-rank 79 Rank-sum 81 Run 81(Sperman s) rank correlation 8367 Chapter 1 PROBABILITY REVIEWB asic CombinatoricsNumber of permutations ofndistinct objects:n!

3 Not all distinct, such as, for exampleaaabbc:6!3!2!1! 63,2,1 orN!n1!n2!n3!..nk! Nn1,n2,n3, .., nk in general, whereN=kPi=1niwhich is the total word length (multinomial coef-ficient).Selectingrout ofnobjects (without duplication), counting all possible arrange-ments:n (n 1) (n 2) .. (n r+1)=n!(n r)! (number of permutations).Forget theirfinal arrangement:Pnrr!=n!(n r)!r! (number of combinations). This will also be called thebinomial we can duplicate (any number of times), and count the arrangements:nrBinomial expansion(x+y)n=nXi=0 ni xn iyiMultinomial expansion(x+y+z)nXi,j,k 0i+j+k=n ni, j, k xiyjzk(x+y+z+w)n=Xi,j,k,c 0i+j+k+c=n ni, j, k, c Experiments (Basic Definitions)Sample spaceis a collection of all possible outcomes of an individual (complete) outcomes are calledsimple the sample space (A, B, C,..).Set TheoryThe old notion of:is (are) now called:Universal set Sample spaceElements of (its individual points )Simple events (complete outcomes)Subsets of EventsEmpty set Null eventWe continue to use the wordintersection(notation:A B, representingthe collection of simple events common to bothAandB),union(A B,simpleeventsbelongingtoeitherAorBorbot h), andcomplement(A,simple eventsnotinA).

4 One should be able to visualize these using Venn diagrams, but whendealing with more than 3 events at a time, one can tackle problems only with thehelp ofBoolean AlgebraBoth and (individually) isdistributiveover union:A (B C ..)=(A B) (A C) ..Similarly, union is distributive over intersection:A (B C ..)=(A B) (A C) ..Trivial rules:A =A, A = ,A A=A, A = ,A =A,A A=A, A A= ,A A= , A= , whenA B(Ais asubsetofB,meaning that every element ofAalsobelongs toB),we get:A B=A(the smaller event) andA B=B(the biggerevent).DeMorgan Laws:A B=A B,andA B=A B,or in generalA B C ..=A B C ..and vice versa ( ).AandBare called (mutually)exclusiveordisjointwhenA B= (nooverlap).Probability of EventsSimple eventscan be assigned aprobability(relative frequency of its occurrencein alongrun). It s obvious that each of these probabilities must be a non-negativenumber.

5 Tofind a probability ofany othereventA(not necessarily simple), wethen add the probabilities of the simple eventsAconsists of. This immediatelyimplies that probabilities must follow a few basic rules:Pr(A) 0Pr( )=0Pr( )=1(the relative frequency of all is obviously1).We should mention thatPr(A)=0does not necessarily imply thatA= .9 Probability rulesPr(A B)=Pr(A)+Pr(B)butonlywhenA B= (disjoint). This implies thatPr(A)=1 Pr(A)as a special also implies thatPr(A B)= Pr(A) Pr(A B).For anyAandB(possibly overlapping) we havePr(A B)=Pr(A)+Pr(B) Pr(A B)Can be extended to:Pr(A B C)=Pr(A)+Pr(B)+Pr(C) Pr(A B) Pr(A C) Pr(B C)+Pr(A B C).IngeneralPr(A1 A2 A3 .. Ak)=kXi=1Pr(Ai) kXi<jPr(Ai Aj)+kXi<j<cPr(Ai Aj Ac) .. Pr(A1 A2 A3 .. Ak)The formula computes the probability thatat least oneof theAievents probability of gettingexactly oneof theAievents is similarly computedby:kXi=1Pr(Ai) 2kXi<jPr(Ai Aj)+3kXi<j<cPr(Ai Aj Ac).

6 KPr(A1 A2 A3 .. Ak)Important resultProbability of any (Boolean) expression involving eventsA, B, C, ..can bealwaysconverted to a linear combination of probabilities of the individual events and theirsimple (non-complemented)intersections(A B, A B C,etc.) treeis a graphical representation of a two-stage (three-stage) random experiment.(effectivelyits sample space - each complete path being a simple event).The individual branch probabilities (usually simple tofigure out), are the socalledconditional rulePr(A B)=Pr(A) Pr(B|A)Pr(A B C)=Pr(A) Pr(B|A) Pr(C|A B)Pr(A B C D)=Pr(A) Pr(B|A) Pr(C|A B) Pr(D|A B C)..Conditional probabilityThe general definition:Pr(B|A) Pr(A B)Pr(A)All basic formulas of probability remain , :Pr(B|A)=1 Pr(B|A),Pr(B C|A)=Pr(B|A)+Pr(C|A) Pr(B C|A), formulaApartitionrepresents chopping the sample space into several smaller events, sayA1,A2,A3.

7 ,Ak,so that they(i)don t overlap ( are all mutually exclusive):Ai Aj= for any1 i, j k(ii)cover the whole ( no gaps ):A1 A2 A3 .. Ak= .For any partition, and an unrelated evenB,we havePr(B)=Pr(B|A1) Pr(A1)+Pr(B|A2) Pr(A2)+..+Pr(B|Ak) Pr(Ak)Independenceof two events is a very natural notion (we should be able to tell from the experi-ment): when one of these events happens, it does not effect the probability of theother. Mathematically, this is expressed by eitherPr(B|A) P(B)or, equivalently, byPr(A B)= Pr(A) Pr(B)Similarly, for three events, theirmutualindependence meansPr(A B C)= Pr(A) Pr(B) Pr(C) independence ofA, B, C, D, ..impliesthat any event build ofA, B,.. must be independent of any event build out ofC, D, .. [as long as the two setsaredistinct].Anotherimportant resultis: To compute the probability of a Boolean ex-pression (itself an event) involving only mutually independent events, it is sufficientto know the events Random VariablesArandom variableyields anumber, for every possible outcome of a (or aformula, calledprobability function) summarizing the in-formation about1.

8 Possible outcomes of the RV (numbers, arranged from the smallest to thelargest)2. the corresponding probabilitiesis called theprobability ,distribution function:Fx(k)=Pr(X k) (joint) distributionoftworandom variablesis similarly specified via the correspondingprobabilityfunctionf(i, j)=Pr(X=i Y=j)with therangeof possibleiandjvalues. Oneofthetworangesisalways marginal (the limits are constant), the other one is conditional ( both of its limits maydepend on the value of the other random variable).Based on this, one can alwaysfind the correspondingmarginal distributionofX:fx(i)=Pr(X=i)=Xj|if(i, j)and, similarly, the marginal distribution distributionofX, given an (observed) value ofY,isdefined byfx(i|Y=j) Pr(X=i|Y=j)=Pr(X=i Y=j)Pr(Y=j)whereivaries over itsconditionalrangeofvalues(givenY=j).Co nditional distribution has all the properties of an ordinary that the outcome ofXcannot influence the outcome ofY(andvice versa) - something we can gather from the implies thatPr(X=i Y=j)=Pr(X=i) Pr(Y=j)foreverypossiblecombination ofiandjMultivariate distributionis a distribution of three of more RVs - conditional distributions can get Value of a RValso called itsmeanoraverage,is a number which corresponds (empirically)to the average value of the random variable when the experiment is repeated,independently, infinitely many times ( it is thelimitof such averages).

9 It iscomputed by x E(X) Xii Pr(X=i)(weighted average), where the summation is over all possible values general,E[g(X)]6=g(E[X]).But, for alinear transformation,E(aX+c)=aE(X)+c12 Expected values related toXandYIngeneralwe haveE[g(X,Y)] =XiXjg(i, j) Pr(X=i Y=j)This would normallynotequal tog( x, y),except:E[aX+bY+c]=aE(X)+bE(Y)+cThe previousformulaeasily extends to any number of variables:E[a1X1+a2X2+..+akXk+c]=a1E(X1) +a2E(X2)+..+akE(Xk)+c(noindependencenece ssary).WhenXandYareindependent,wealsohav eE(X Y)=E(X) E(Y)and, ingeneral:E[g1(X) g2(Y)] =E[g1(X)] E[g2(Y)]Moments (univariate)Simple:E(Xn)Central:E[(X x)n]Of these, the most important is thevarianceofX:E (X x)2 =E(X2) x2 Itssquarerootisthestandard deviationofX,notation: x=pVar(X)(thisis the Greek letter sigma ).The interval to + should contain the bulk of the distribution anywhere from50to90%.

10 WhenY aX+c(a linear transformation ofX),we getVar(Y)=a2 Var(X)which implies y=|a| xMoments (bivariate or joint )Simple:E(Xn Ym)CentralE (X x)n (Y y)m The most important of these is thecovarianceofXandY:Cov(X,Y) E (X x) (Y y) E(X Y) x y13It becomeszerowhenXandYareindependent, but: zero covariance doesnotnecessarily imply related quantity is the correlation coefficient betweenXandY: xy=Cov(X,Y) x y(this is the Greek letter rho ). The absolute value of this coefficientcannotbegreater than ofaX+bY+cis equal toa2 Var(X)+b2 Var(Y)+2abCov(X, Y)Independencewould make the last a linear combination ofany numberof random variables:Var(a1X1+a2X2+..akXk+c)=a21 Var(X1)+a22 Var(X2)+..+a2kVar(Xk)+2a1a2 Cov(X1,X2)+2a1a3 Cov(X1,X3)+..+2ak 1akCov(Xk 1,Xk)Moment generating functionis defined byMx(t) E etX wheretis an arbitrary (real) (Xk)=M(k)x(t=0)or, in words, to get thekthsimple moment differentiate the correspondingMGFktimes (with respect tot)and settequal to For twoindependentRVs we have:MX+Y(t)=MX(t) MY(t)This result can be extended toany numberof mutually independent RVs:MX+Y+Z(t)=MX(t) MY(t) MZ(t) And,finallyMaX+c(t)=ect MX(at)Probability generating functionis defined byPx(s)=Pr(X=0)+Pr(X=1)s+Pr(X=2)s2+Pr(X= 3)s3+.