ADVANCED PROBABILITY AND STATISTICAL INFERENCE I

1 ADVANCED PROBABILITY ANDSTATISTICAL INFERENCE ILecture Notes of BIOS 760 Distribution of Normalized Summation Uniform Random VariablesPREFACET hese course notes have been revised based on my past teaching experience at the departmentof Biostatistics in the University of North Carolina in Fall 2004 and Fall 2005. The context in-cludes distribution theory , PROBABILITY and measure theory , large sample theory , theory of pointestimation and efficiency theory . The last chapter specially focuses on maximum likelihoodapproach. Knowledge of fundamental real analysis and STATISTICAL INFERENCE will be helpful forreading these parts of the notes are compiled with moderate changes based on two valuable textbooks: theory of Point Estimation(second edition, Lehmann and Casella, 1998) andA Course inLarge Sample theory (Ferguson, 2002).

2 Some notes are also borrowed from a similar coursetaught in the University of Washington, Seattle, by Professor Jon Wellner. The revision hasincorporated valuable comments from my colleagues and students sitting in my previous , there are inevitably numerous errors in the notes and I take all the responsibilitiesfor these ZengAugust, 2006 CHAPTER 1 A REVIEW OFDISTRIBUTION THEORYThis chapter reviews some basic concepts of discrete and continuous random variables. Distri-bution results on algebra and transformations of random variables (vectors) are given. Part ofthe chapter pays special attention to the properties of the Gaussian distributions. The finalpart of this chapter introduces some commonly-used distribution Basic ConceptsRandom variables are often classified intodiscrete random variablesandcontinuous randomvariables.

3 By names, discrete random variables are some variables which take discrete valueswith an associatedprobability mass function; while, continuous random variables are variablestaking non-discrete values (usuallyR) with an associatedprobability density function. A proba-bility mass function consists of countable non-negative values with their total sum being one anda PROBABILITY density function is a non-negative function in real line with its whole integrationbeing , the above definitions are not rigorous. What is the precise definition of a randomvariable? Why shall we distinguish between mass functions or density functions? Can somerandom variable be both discrete and continuous? The answers to these questions will becomeclear in next chapter on PROBABILITY measure theory .

4 However, you may take a glimpse below:(a) Random variables are essentiallymeasurable functionsfrom aprobability measure spaceto real space. Especially, discrete random variables map into discrete set and continuousrandom variables map into the whole real line.(b) PROBABILITY ( PROBABILITY measure) is a function assigning non-negative values to sets of a -fieldand it satisfies the property ofcountable additivity.(c) PROBABILITY mass function for a discrete random variable is theRadon-Nykodym derivativeofrandom variable-induced measurewith respect to acounting measure. Probabilitydensity function for continuous random variable is the Radon-Nykodym derivative ofrandom variable-induced measure with respect to theLebesgue this chapter, we do not need to worry about these abstract quantities to describe the distribution of a random variable includecumulative distri-bution function,mean,variance,quantile,mode,mom ents,centralized moments,kurtosisandskewness.

5 For instance, ifXis a discrete random variable taking valuesx1,x2,..with probabili-tiesm1,m2,.. The cumulative distribution function ofXis defined asFX(x) = xi xmi. The1 DISTRIBUTION THEORY2kth moment ofXis given asE[Xk] = imixkiand thekth centralized moment ofXis given asE[(X )k] where is the expectation ofX. IfXis a continuous random variable with prob-ability density functionfX(x), then the cumulative distribution functionFX(x) = x fX(t)dtand thekth moment ofXis given asE[Xk] = xkfX(x)dxif the integration is skewness ofXis given byE[(X )3]/V ar(X)3/2and the kurtosis ofXis given byE[(X )4]/V ar(X)2. The last two quantities describe the shape of the density function:negative values for the skewness indicate the distribution that are skewed left and positive val-ues for the skewness indicate the distribution that are skewed right.

6 By skewed left, we meanthat the left tail is heavier than the right tail. Similarly, skewed right means that the righttail is heavier than the left tail. Large kurtosis indicates a peaked distribution and smallkurtosis indicates a flat distribution. Note that we have already usedE[g(X)] to denote theexpectation ofg(X). Sometimes, we use g(x)dFX(x) to represent it no matter wetherXiscontinuous or discrete. This notation will be clear after we introduce the PROBABILITY we review an important definition in distribution theory , namely thecharacteris-tic functionofX. By definition, the characteristic function forXis defined as X(t) =E[exp{itX}] = exp{itx}dFX(x), whereiis the imaginary unit, the square-root of -1.

7 Equiva-lently, X(t) is equal to exp{itx}fX(x)dxfor continuousXand is jmjexp{itxj}for discreteX. The characteristic function is important since it uniquely determines the distribution func-tion ofX, the fact implied in the following (Uniqueness Theorem)If a random variableXwith distribution functionFXhas a characteristic function X(t) and ifaandbare continuous points ofFX, thenFX(b) FX(a) = limT 12 T Te ita e itbit X(t) , ifFXhas a density functionfX(for continuous random variableX) , thenfX(x) =12 e itx X(t)dt. We defer the proof to Chapter 3. Similar to the characteristic function, we can define themoment generating functionforXasMX(t) =E[exp{tX}]. However, we note thatMX(t) maynot exist for sometbut X(t) always important and distinct feature in distribution theory is the independence of tworandom variables.

8 For two random variablesXandY, we sayXandYareindependentifP(X x,Y y) =P(X x)P(Y y); , the joint distribution function of (X,Y)is the product of the two marginal distributions. If (X,Y) has a joint density, then anequivalent definition is that the joint density of (X,Y) is the product of two marginal den-sities. Independence introduces many useful properties, among which one important propertyis thatE[g(X)h(Y)] =E[g(X)]E[h(Y)] for any sensible functionsgandh. In more gen-eral case whenXandYmay not be independent, we can calculate theconditional densityofXgivenY, denoted byfX|Y(x|y), as the ratio between the joint density of (X,Y) andthe marginal density ofY. Thus, the conditional expectation ofXgivenY=yis equal toDISTRIBUTION THEORY3E[X|Y=y] = xfX|Y(x|y)dx.

9 Clearly, whenXandYare independent,fX|Y(x|y) =fX(x)andE[X|Y=y] =E[X]. For conditional expectation, two formulae are useful:E[X] =E[E[X|Y]] andV ar(X) =E[V ar(X|Y)] +V ar(E[X|Y]).So far, we have reviewed some basic concepts for a single random variable. All the abovedefinitions can be generalized to multivariate random vectorX= (X1,..,Xk) with a jointprobability mass function or a joint density function. For example, we can define the meanvector ofXasE[X] = (E[X1],..,E[Xk]) and define the covariance matrix forXasE[XX ] E[X]E[X] . The cumulative distribution function forXis ak-variate functionFX(x1,..,xk) =P(X1 x1,..,Xk xk) and the characteristic function ofXis ak-variate function, defined as X(t1,..,tk) =E[ei(t1X1+..+tkXk)] = Rkei(t1x1+.)

10 +tkxk)dFX(x1,..,xk).Same as Theorem , an inversion formula holds: LetA={(x1,..,xk) :a1< x1 b1,..,ak<xk bk}be a rectangle inRkand assumeP(X A) = 0, where Ais the boundary (b1,..,bk) FX(a1,..,ak) =P(X A)= limT 1(2 )k T T T Tk j=1e itjaj e itjbjitj X(t1,..,tk)dt1 , we can define the conditional density, the conditional expectation, the independence oftwo random vectors similar to the univariate Examples of Special DistributionsWe list some commonly-used distributions in the following Bernoulli Distribution and Binomial DistributionA random variableXis said to be Bernoulli(p) ifP(X= 1) =p= 1 P(X= 0). IfX1,..,Xnare independent,identically distributed ( ) Bernoulli(p), thenSn=X1+..+Xnhas a binomial distribution,denoted bySn Binomial(n,p), withP(Sn=k) =(nk)pk(1 p)n mean ofSnis equal tonpand the variance ofSnis equal tonp(1 p).