University of Toronto

1 Probability and StatisticsThe Science of UncertaintySecond EditionMichael J. Evans and Je rey S. RosenthalUniversity of TorontoContentsPrefaceix1 Probability Probability: A Measure of Uncertainty .. Why Do We Need Probability Theory?.. Probability VennDiagramsandSubsets .. Properties of Probability Models .. Uniform Probability on Finite Spaces .. Conditional Probability and Independence .. Conditional Probability .. IndependenceofEvents .. Continuity FurtherProofs(Advanced) .. 312 Random Variables and CdfsofDiscreteDistributions .. CdfsofAbsolutelyContinuousDistributions .. Distributions Neither Discrete Nor Continuous (Advanced) .. One-DimensionalChangeofVariable .. TheDiscreteCase .. Joint Probability Functions .. JointDensityFunctions .. ConditioningandIndependence .. Conditioning on Discrete Random Variables.

2 Conditioning on Continuous Random Variables .. IndependenceofRandomVariables .. Multidimensional Change of TheDiscreteCase .. TheContinuousCase(Advanced) .. Simulating Probability Distributions .. SimulatingDiscreteDistributions .. (Advanced) .. 1253 Variance,Covariance,andCorrelation .. CharacteristicFunctions(Advanced).. ConditionalExpectation .. Conditional Variance (Advanced) .. Inequalities .. Jensen sInequality(Advanced) .. GeneralExpectations(Advanced) .. FurtherProofs(Advanced) .. 1944 Sampling Distributions and SamplingDistributions .. Convergence in Probability .. Convergence with Probability 1 .. TheStrongLawofLargeNumbers .. ConvergenceinDistribution .. TheCentralLimitTheorem .. The Central Limit Theorem and Assessing Error .. NormalDistributionTheory .. TheChi-SquaredDistribution .. FurtherProofs(Advanced).

3 2465 Statistical WhyDoWeNeedStatistics?.. Inference Using a Probability Model .. FinitePopulations .. SimpleRandomSampling .. SomeBasicInferences .. PlottingData .. TypesofInference .. 2896 Likelihood TheLikelihoodFunction .. Maximum Likelihood Estimation .. The Multidimensional Case (Advanced).. StandardErrors,Bias,andConsistency .. ConfidenceIntervals .. TestingHypothesesandP-Values .. Sample-Size Calculations: Sample-SizeCalculations:Power .. Distribution-FreeMethods .. MethodofMoments .. The Sign Statistic and Inferences about Quantiles .. LargeSampleBehavioroftheMLE(Advanced).. 3647 Bayesian Sampling from the Posterior Using Gibbs Sampling(Advanced).. ConjugatePriors .. EmpiricalBayes .. FurtherProofs(Advanced) .. Derivation of the Posterior Distribution for the Location-ScaleNormalModel.

4 Derivation ofJ( ( 0, ))for the Location-Scale Normal .. 4318 Optimal Optimal Unbiased Estimation .. The Rao Blackwell Theorem and Rao Blackwellization .. Completeness and the Lehmann Scheff Theorem .. The Cramer Rao Inequality (Advanced) .. Optimal Hypothesis Testing .. ThePowerFunctionofaTest .. TheNeyman PearsonTheorem .. LikelihoodRatioTests(Advanced) .. Optimal Bayesian Inferences .. DecisionTheory(Advanced).. FurtherProofs(Advanced) .. 4739 Model CheckingtheSamplingModel .. Residual and Probability PredictionandCross-Validation .. WhatDoWeDoWhenaModelFails?.. Checking for Prior Data The Problem with Multiple Checks .. 50910 Relationships Among .. The Cause EffectRelationshipsandExperiments .. RandomPredictor .. BayesianFormulation .. Quantitative Response and Predictors .. TheMethodofLeastSquares .. TheSimpleLinearRegressionModel.

5 BayesianSimpleLinearModel(Advanced).. The Multiple Linear Regression Model (Advanced) .. Quantitative Response and Categorical Predictors .. OneCategoricalPredictor(One-WayANOVA) .. RepeatedMeasures(PairedComparisons).. TwoCategoricalPredictors(Two-WayANOVA) .. RandomizedBlocks .. One Categorical and One Quantitative Categorical Response and Quantitative Predictors .. (Advanced) .. 60711 Advanced Topic Stochastic TheDistributionoftheFortune .. TheGambler sRuinProblem .. StationaryDistributions .. Markov Chain Limit Theorem .. TheMetropolis .. Definition of a Martingale .. ExpectedValues .. Brownian Motion as a Limit .. DiffusionsandStockPrices .. 668 Appendices675A Mathematical Derivatives .. Matrix Multiplication .. MultivariableIntegrals .. 679B 699C Common 706D Chi-Squared Distribution Quantiles .. Quantiles.

6 Binomial Distribution Probabilities .. 724E Answers to Odd-Numbered Exercises729 Index750 PrefaceThis book is an introductory text on probability and statistics , targeting students whohave studied one year of calculus at the University level and are seeking an introductionto probability and statistics with mathematical content. Where possible, we providemathematical details, and it is expected that students are seeking to gain some masteryover these, as well as to learn how to conduct data analyses. All the usual method-ologies covered in a typical introductory course are introduced, as well as some of thetheory that serves as their text can be used with or without a statistical computer package. It is our opin-ion that students should see the importance of various computational techniques inapplications, and the book attempts to do this. Accordingly, we feel that computationalaspects of the subject, such as Monte Carlo, should be covered, even if a statisticalpackage is not used.

7 Almost any statistical package is suitable. AComputationsappendix provides an introduction to the R language. This covers all aspects of thelanguage needed to do the computations in the text. Furthermore, we have providedthe R code for any of the more complicated computations. Students can use theseexamples as templates for problems that involve such computations, for example, us-ing Gibbs sampling. Also, we have provided, in a separate section of this appendix,Minitab code for those computations that are slightly involved, , Gibbs programming experience is required of students to do the have organized the exercises in the book into groups, as an aid to suitable for all students and offer practice in applying the concepts discussedin a particular greater understanding, and a student can ex-pect to spend more thinking time on these. If a problem is marked (MV), then it willrequire some facility with multivariable calculus beyond thefirst calculus course, al-though these problems are not necessarily problems that moststudents willfind difficult;these are only for students who have no trouble with theExercisesand theProblems.

8 There are alsoComputer ExercisesandComputerProblems, where it is expected that students will make use of a statistical package inderiving have included a number ofDiscussion Topicsdesigned to promote criticalthinking in students. Throughout the book, we try to point students beyond the masteryof technicalities to think of the subject in a larger frame of reference. It is important thatstudents acquire a sound mathematical foundation in the basic techniques of probabilityand statistics , which we believe this book will help students accomplish. Ultimately,however, these subjects are applied in real-world contexts, so it is equally importantthat students understand how to go about their application and understand what issuesarise. Often, there are no right answers toDiscussion Topics;their purpose is to get axPrefacestudent thinking about the subject matter. If these were to be used for evaluation, thenthey would be answered in essay format and graded on the maturity the student showedwith respect to the issues Topicsare probably most suitable forsmaller classes, but these will also benefit students who simply read them over andcontemplate their sections of the book are labelledAdvanced.

9 This material is aimed at stu-dents who are more mathematically mature (for example, they are taking, or have taken,a second course in calculus). All theAdvancedmaterial can be skipped, with no lossof continuity, by an instructor who wishes to do so. In particular, thefinal chapter of thetext is labelledAdvancedand would only be taught in a high-level introductory courseaimed at specialists. Also, many proofs appear in thefinal section of many chapters,labelledFurther Proofs (Advanced). An instructor can choose which (if any) of theseproofs they wish to present to their such, we feel that the material in the text is presented in aflexible way thatallows the instructor tofind an appropriate level for the students they are teaching. AMathematical Backgroundappendix reviews some mathematical concepts, from afirst course in calculus, in case students coulduse a refresher, as well as brief introduc-tions to partial derivatives, double integrals, 1introduces the probability model and provides motivation for the studyof probability.

10 The basic properties of a probability measure are 2deals with discrete, continuous, joint distributions, and the effects ofa change of variable. It also introduces the topic of simulating from a probabilitydistribution. The multivariate change of variable is developed in an Advanced 3introduces expectation. The probability-generating function is dis-cussed, as are the moments and the moment-generating function of a random chapter develops some of the major inequalities used in probability. A section oncharacteristic functions is included as an Advanced 4deals with sampling distributions and limits. Convergence in probabil-ity, convergence with probability 1, the weak and strong laws of large numbers, con-vergence in distribution, and the central limit theorem are all introduced, along withvarious applications such as Monte Carlo. The normal distribution theory, necessaryfor many statistical applications, is also dealt with mentioned, Chapters 1 through 4 include material on Monte Carlo is a key aspect of the application of probability theory, and it is our viewthat its teaching should be integrated with the theory right from the start.