Review of basic probability and statistics - U-M LSA ...

1 Review of basic probability and statisticsProbability: basic definitions Arandom variableis the outcome of a natural process that can not be predicted withcertainty. Examples: the maximum temperature next Tuesday in Chicago, the price of Wal-Mart stock two days from now, the result of flipping a coin, the response of apatient to a drug, the number of people who will vote for a certain candidate ina future election. On the other hand, the time of sunrise next Tuesday is for all practical purposesexactly predictable from the laws of physics, and hence is not really a randomvariable (although technically it may be called adegeneraterandom variable). There is some grayness in this definition: eventually we may be able to predictthe weather or even sociological phenomena like voting patterns with extremelyhigh precision. From a practical standpoint this is not likely to happen any timesoon, so we consider a random variable to be the state of a natural process thathuman beings cannot currently predict with certainty.

2 The set of all possible outcomes for a random variable is called thesample space. Corre-sponding to each point in the sample space is aprobability, which is a number between0 and 1. The sample space together with all probabilities is called thedistribution. Properties of probabilities: (i) a probability is always a number between 0 and 1, (ii)the sum of probabilities for all points in the samples space is always exactly 1. Example:IfXis the result of flipping a fair coin, the sample space ofXis{H, T}(Hfor heads,Tfor tails). Either outcome has probability 1/2, so wewriteP(X=H) = 1/2 ( the probability thatXis a head is 1/2) andP(X=T) = 1/2. The distribution can be written{H 1/2, T 1/2}. Example:IfXis the number of heads observed in four flips of a fair coin, thesample space ofXis{0,1,2,3,4}. The probabilities are given by the binomialdistribution. The distribution is{0 1/16,1 1/4,2 3/8,3 1/4,4 1/16}.

3 Example:Suppose we select a point on the surface of the Earth at random andmeasure the temperature at that point with an infinitely precise temperature will certainly fall between 100 Cand 100 C, but there areinfinitely many values in that range. Thus we can not represent the distributionusing a list{x y, ..}, as above. Solutions to this problem will be discussedbelow. A random variable is eitherqualitativeorquantitativedepending on the type of valuein the sample space. Quantitative random variables express values like temperature,mass, and velocity. Qualitative random variables express values like gender and Thecumulative distribution function(CDF)is a way to represent a quantitative distri-bution. For a random variableX, the CDF is a functionF(t) such thatF(t) =P(X t). That is, the CDF is a function oftthat specifies the probability of observing avalue no larger thant. Example:SupposeXfollows a standard normal distribution.

4 You may recallthat this distribution has median 0, so that theP(X 0) = 1/2 andP(X 0) = 1/2. Thus for the standard normal distribution,F(0) = 1/2. There is nosimple formula forF(t) whent6= 0, but a table of values forF(t) is found in theback of almost any statistics textbook. A plot ofF(t) is shown below. 0 1-4-3-2-1 0 1 2 3 4P(X <= t)tThe standard normal CDF Any CDFF(t) has the following properties: (i) 0 F(t) 1, (ii)F( ) = 0, (iii)F( ) = 1, (iv)Fis We can read probabilities of the formP(X t) directly from the graph of the CDF. SinceP(X > t) = 1 P(X t) = 1 F(t), we can also read off a probability of the formP(X > t)directly from a graph of the CDF. 0 1-4-3-2-1 0 1 2 3 4 The length of the green line is the probability of observing a value less than 1. The length ofthe blue line is the probability of observing a value greater than1. The length of the purpleline is the probability of observing a values less Ifa b, for any random variableXP(a < X b) =P(X b) P(X a) =F(b) F(a).

5 Thus we can easily determine the probability of observing a value in an interval (a, b] fromthe CDF. 0 1-4-3-2-1 0 1 2 3 4 The length of the purple line is the probability of observing a value between Ifaandbfall in an area whereFis very steep,F(b) F(a) will be relatively we are more likely to observe values whereFis steep than whereFis flat. Aprobability density function (PDF)is a different way to represent a probability dis-tribution. The PDF forXis a functionf(x) such that the probability of observing avalue ofXbetweenaandbis equal to the area under the graph off(x) betweenaandb. A plot off(x) for the standard normal distribution is shown below. We are morelikely to observed values wherefis large than wherefis 0 0 1 2 3 4 DensityValueThe standard normal PDF5 probability : samples and populations If we can repeatedly and independently observe a random variablentimes, we haveanindependent and identically distributed sample of sizen, or aniidsample of is also called asimple random sample, or anSRS.)

6 (Note that the wordsampleisbeing used somewhat differently in this context compared to its use in the termsamplespace). A central problem in statistics is to answer questions about an unknown distributioncalled thepopulationbased on a simple random sample that was generated by thedistribution. This process is , given a numerical characteristic of a distribution, we may wish toestimatethe value of that characteristic based on data. In an iid sample, each point in the sample space will be observed with a certainfrequency. For example, if we flip a fair coin 20 times we might observe 13 heads,so the frequency of heads is 13/20. Due to random variation, this frequency differssomewhat from the underlying probability , which is 1/2. If the sample is sufficientlylarge, frequencies and probabilities will be very similar (this is known as thelaw oflarge numbers). Since probabilities can be estimated as frequencies, and the CDF is defined in terms ofprobabilities ( (t) =P(X t)), we can estimate the CDF as theempirical CDF(ECDF).

7 Suppose thatX1, X2, .. , Xnare an iid sample. Then the ECDF (evaluatedatt) is defined to be the proportion of theXithat are not larger thant. The ECDF isnotated as F(t) (in general the symbol ?represents an estimate based on an iid sampleof a characteristic of the population named?). Example:Suppose we observe a sample of sizen= 4 whose sorted values are3,5,6,11. Then F(t) is equal to: 0 fort <3, 1/4 for 3 t <5, 1/2 for 5 t <6,3/4 for 6 t <11, and 1 fort 0 1 0 2 4 6 8 10 12 14 16P(X<=t)tThe ECDF for the data set{3, 5, 6, 11} Since the ECDF is a function of the sample, which is random, if we construct twoECDF s for two samples from the same distribution, the results will differ (even throughthe CDF s from the underlying population are the same). This is calledsampling vari-ation. The next figure shows two ECDF s constructed from two independent samplesof size 50 from a standard normal population.

8 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDF 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDFTwo ECDF s for standard normal samples of size 50 (the CDF is shown in red) The sampling variation gets smaller as the sample size increases. The following figureshows ECDF s based on SRS s of sizen= 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDF 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDFTwo ECDF s for standard normal samples of size 500 (the CDF is shown in red) The sampling variation gets larger as the sample size decreases. The following figureshows ECDF s based on SRS s of sizen= 10. 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDF 0 1-4-3-2-1 0 1 2 3 4P(X<=t)tCDFECDFTwo ECDF s for standard normal samples of size 10 (the CDF is shown in red)8 Given a SRSX1, .. , Xn, ahistogramformed from the SRS is an estimate of the construct a histogram, select abin width >0, and letH(x) be the function suchthat when (k 1) x < k ,H(x) is the number of observedXithat fall between(k 1) andk.

9 To directly compare a density and a histogram they must be put on the same density is based on a sample of size 1, so to compare it to a histogram based onnobservations using bins with width , the density must be scaled by n. There is no single best way to select . A rule of thumb for the number of bins is =Rlog2(n) + 1,wherenis the number of data points andRis therangeof the data (the greatest valueminus the least value). This can be used to produce a reasonable value for . Just as with the ECDF, sampling variation will cause the histogram to vary if the experimentis repeated. The next figure shows two replicates of a histogram generated from an SRS of50 standard normal random draws. 0 5 10 15 20-4-3-2-1 0 1 2 3 4f(x)xScaled densityHistogram 0 2 4 6 8 10 12 14 16-4-3-2-1 0 1 2 3 4f(x)xScaled densityHistogramTwo histograms for standard normal samples of size 50 (the scaled density isshown in red) As with the ECDF, larger sample sizes lead to less sampling variation.

10 This is illus-trated in comparing the previous figure to the next 0 20 40 60 80 100 120 140-4-3-2-1 0 1 2 3 4f(x)xScaled densityHistogram 0 20 40 60 80 100 120 140-4-3-2-1 0 1 2 3 4f(x)xScaled densityHistogramTwo histograms for standard normal samples of size 500 (the scaled density isshown in red) Thequantile functionis the inverse of the CDF. It is the functionQ(p) such thatF(Q(p)) =P(X Q(p)) =p,where 0 p 1. In words,Q(p) is the point in the sample space such that withprobabilitypthe observation will be less than or equal toQ(p). For example,Q(1/2)is the median:P(X Q(1/2)) = 1/2, and the 75thpercentile isQ(3/4). A plot of the quantile function is just a plot of the CDF with thexandyaxes the CDF, the quantile function is 0 1 2 3 4 0 1t: P(X <= t) = ppThe standard normal quantile function Suppose we observe an SRSX1, X2, .. , Xn. Sort these values to giveX(1) X(2) X(n)(these are called theorder statistics ).