Random Variables and Probability Distributions

1 ARandom Variablesand Probability Distribution Functions and Random The Multivariate Normal Distribution Functions and ExpectationThe distribution functionFof a Random variableXis defined byF(x)=P[X x]( )for all realx. The following properties are direct consequences of ( ) nondecreasing, ,F(x) F(y)ifx right continuous, ,F(y) F(x)asy (x) 1andF(y) 0asx andy , , any function that satisfies properties 1 3 is the distribution function ofsome Random of the commonly encountered distribution functionsFcan be expressedeither asF(x)= x f(y)dy( )orF(x)= j:xj xp(xj),( )where{x0,x1,x2,..}is a finite or countably infinite set. In the case ( )weshallsay that the Random variableXiscontinuous. The functionfis called theprobabilitydensity function(pdf) ofXand can be found from the relationf(x)=F (x). Springer International Publishing Switzerland Brockwell, Davis,Introduction to Time Series and Forecasting,Springer Texts in Statistics, DOI A Random Variables and Probability DistributionsIn case ( ), the possible values ofXare restricted to the set{x0,x1.}

2 },andweshall say that the Random variableXisdiscrete. The functionpis called theprobabilitymass function(pmf) ofX,andFis constant except for upward jumps of sizep(xj)atthe pointsxj. Thusp(xj)is the size of the jump inFatxj, ,p(xj)=F(xj) F(x j)=P[X=xj],whereF(x j)=limy xjF(y). Examples of Continuous Distributions (a)The normal distribution with mean and variance say that a randomvariableXhas the normal distribution with mean and variance 2 written moreconcisely asX N , 2 ifXhas the pdf given byn x; , 2 =(2 ) 1/2 1e (x )2/(2 2) <x< .It follows then thatZ=(X )/ N(0,1)and thatP[X x]=P Z x = x ,where (x)= x (2 ) 1/2e 12z2dzis known as thestandard normal distribu-tion function. The significance of the termsmeanandvariancefor the parameters and 2is explained below (see ).(b)The uniform distribution on[a,b].The pdf of a Random variable uniformly dis-tributed on the interval[a,b]is given byu(x;a,b)= 1b a,ifa x b,0,otherwise.

3 (c)The exponential distribution with parameter .The pdf of an exponentially dis-tributed Random variable with parameter >0ise(x; )= 0,ifx<0, e x,ifx corresponding distribution function isF(x)= 0,ifx<0,1 e x,ifx 0.(d)The gamma distribution with parameters and .The pdf of a gamma-distributedrandom variable isg(x; , )= 0,ifx<0,x 1 e x/ ( ),ifx 0,where the parameters and are both positive and is the gamma functiondefined as ( )= 0x 1e Distribution Functions and Expectation355 Note thatfis the exponential pdf when =1and that when is a positive integer ( )=( 1)!with0!defined to be1.(e)The chi-squared distribution with degrees of each positive integer , the chi-squared distribution with degrees of freedom is defined to be thedistribution of the sumX=Z21+ +Z2 ,whereZ1,..,Z are independent normally distributed Random Variables withmean 0 and variance 1.

4 This distribution is the same as the gamma distributionwith parameters = /2and = Examples of Discrete Distributions (f)The binomial distribution with parameters n and pmf of a binomiallydistributed Random variableXwith parametersnandpisb(j;n,p)=P[X=j]= nj pj(1 p)n j,j=0,1,..,n,wherenis a positive integer and0 p 1.(g)The uniform distribution on{1,2,..,k}.The pmf of a Random variableXuni-formly distributed on{1,2,..,k}isp(j)=P[X=j]=1k,j=1, ,k,wherekis a positive integer.(h)The Poisson distribution with parameter .A Random variableXissaidtohaveaPoisson distribution with parameter >0ifp(j; )=P[X=j]= jj!e ,j=0,1,..We shall see in that is the mean ofX.(i)The negative binomial distribution with parameters and Random variableXis said to have a negative binomial distribution with parameters >0andp [0,1]if it has pmfnb(j; ,p)= j k=1k 1+ k (1 p)jp ,j=0,1,..,where the product is defined to be 1 ifj= all Random Variables can be neatly categorized as either continuous or example, consider the time you spend waiting to be served at a checkout counterand suppose that the Probability of finding no customers ahead of you is12.

5 Then thetime you spend waiting for service can be expressed asW= 0,with probability12,W1,with probability12,356 Appendix A Random Variables and Probability DistributionswhereW1is a continuous Random variable . If the distribution ofW1is exponential withparameter 1, then the distribution function ofWisF(x)= 0,ifx<0,12+12 1 e x =1 12e x,ifx distribution function is neither continuous (since it has a discontinuity atx=0)nor discrete (since it increases continuously forx>0). It is expressible as amixture,F=pFd+(1 p)Fc,withp=12, of a discrete distribution functionFd= 0,x<0,1,x 0,and a continuous distribution functionFc= 0,x<0,1 e x,x distribution function can in fact be expressed in the formF=p1Fd+p2Fc+p3 Fsc,where0 p1,p2,p3 1,p1+p2+p3=1,Fdis discrete,Fcis continuous, andFscissingular continuous(continuous but not of the ). Distribution functionswith a singular continuous component are rarely Expectation, Mean, and VarianceTheexpectationof a functiongof a Random variableXis defined byE(g(X))= g(x)dF(x),where g(x)dF(x):= g(x)f(x)dxin the continuous case, j=0g(xj)p(xj)in the discrete case,andgis any function such thatE|g(x)|<.

6 (IfFis the mixtureF=pFc+(1 p)Fd,thenE(g(X))=p g(x)dFc(x)+(1 p) g(x)dFd(x).) ThemeanandvarianceofXare defined as =EXand 2=E(X )2, respectively. They are evaluated bysettingg(x)=xandg(x)=(x )2in the definition ofE(g(X)).It is clear from the definition that expectation has thelinearity propertyE(aX+b)=aE(X)+bfor any real constantsaandb(provided thatE|X|< ). Random Vectors357 Example Normal DistributionIfXhas the normal distribution with pdfn x; , 2 as defined in Example (a) above,thenE(X )= (x )n x; , 2 dx= 2 n x: , 2 dx= shows, with the help of the linearity property ofE,thatE(X)= , , that the parameter isin fact the mean of the normal distribution defined inExample (a). Similarly,E(X )2= (x )2n x; , 2 dx= 2 (x )n x; , 2 by parts and using the fact thatfis a pdf, we find that the variance ofXisE(X )2= 2 n x; , 2 dx= 2. Example Poisson DistributionThe mean of the Poisson distribution with parameter (see Example (h) above) isgiven by = j=0j jj!

7 E = j=1 j 1(j 1)!e = e e = .A similar calculation shows that the variance is also equal to (see Problem ). and parameters associated with a Random variableXwill belabeled with the subscriptXwhenever it is necessary to identify the particular randomvariable to which they refer. For example, the distribution function, pdf, mean, andvariance ofXwill be written asFX,fX, X,and 2X, respectively, whenever it isnecessary to distinguish them from the corresponding quantitiesFY,fY, Y,and 2 Yassociated with a different Random Random VectorsAnn-dimensional Random vector is a column vectorX=(X1,..,Xn) each of whosecomponents is a Random variable . The distribution functionFofX, also called thejoint distributionofX1,..,Xn, is defined byF(x1,..,xn)=P[X1, x1,..,Xn xn]( )for all real numbersx1,..,xn. This can be expressed in a more compact form asF(x)=P[X x],x=(x1,..,xn) ,for all real vectorsx=(x1.)

8 ,xn) . The joint distribution of any subcollectionXi1,..,Xikof these Random Variables can be obtained fromFby settingxj= 358 Appendix A Random Variables and Probability Distributionsin ( )forallj/ {i1,..,ik}. In particular, the Distributions ofX1and(X1,Xn) aregiven byFX1(x1)=P[X1 x1]=F(x1, ,.., )andFX1,Xn(x1,xn)=P[X1 x1,Xn xn]=F(x1, ,.., ,xn).As in the univariate case, a Random vector with distribution functionFis said to becontinuous ifFhas a density function, , ifF(x1,..,xn)= xn x2 x1 f(y1,..,yn)dy1dy2 Probability density ofXis then found fromf(x1,..,xn)= nF(x1,..,xn) x1 Random vectorXis said to be discrete if there exist real-valued vectorsx0,x1,..and a Probability mass functionp(xj)=P[X=xj]such that j=0p(xj)= expectation of a functiongof a Random vectorXis defined byE(g(X))= g(x)dF(x)= g(x1,..,xn)dF(x1,..,xn),where g(x1,..,xn)dF(x1,..,xn)= g(x1.

9 ,xn)f(x1,..,xn)dx1 dxn,in the continuous case, j1 jng(xj1,..,xjn)p(xj1,..,xjn),in the discrete case,andgis any function such thatE|g(X)|< .The Random variablesX1,..,Xnare said to beindependentifP[X1 x1,..,Xn xn]=P[X1 x1] P[Xn xn], ,F(x1,..,xn)=FX1(x1) FXn(xn)for all real numbersx1,..,xn. In the continuous and discrete cases, independence isequivalent to the factorization of the joint density function or Probability mass functioninto the product of the respective marginal densities or mass functions, ,f(x1,..,xn)=fX1(x1) fXn(xn)( )orp(x1,..,xn)=pX1(x1) pXn(xn).( )For two Random vectorsX=(X1,..,Xn) andY=(Y1,..,Ym) with jointdensity functionfX,Y, the conditional density ofYgivenX= Random Vectors359fY|X(y|x)= fX,Y(x,y)fX(x),iffX(x)>0,fY(y),iffX(x)= conditional expectation ofg(Y)givenX=xis thenE(g(Y)|X=x)= g(y)fY|X(y|x) independent, thenfY|X(y|x)=fY(y)by ( ), and so the conditionalexpectation ofg(Y)givenX=xisE(g(Y)|X=x)=E(g(Y)),whic h, as expected, does not depend onx.

10 The same ideas hold in the discrete casewith the Probability mass function assuming the role of the density Means and CovariancesIfE|Xi|< for eachi, then we define the mean or expected value ofX=(X1,..,Xn) to be the column vector X=EX=(EX1,..,EXn) .In the same way we define the expected value of any array whose elements are randomvariables ( , a matrix of Random Variables ) to be the same array with each randomvariable replaced by its expected value (if the expectation exists).IfX=(X1,..,Xn) andY=(Y1,..,Ym) are Random vectors such that eachXiandYjhas a finite variance, then thecovariance matrixofXandYis defined to bethe matrix XY=Cov(X,Y)=E[(X EX)(Y EY) ]=E(XY ) (EX)(EY) .The(i,j)element of XYis the covariance Cov(Xi,Yj)=E(XiYj) E(Xi)E(Yj).Inthespecial case whereY=X,Cov(X,Y)reduces to the covariance matrix of the suppose thatYandXare linearly related through the equationY=a+BX,whereais anm-dimensional column vector andBis anm nmatrix.