2.2.3 Minimum Variance Unbiased Estimators

1 POINT Minimum Variance Unbiased EstimatorsIf an Unbiased estimator has the Variance equal to the CRLB, it must have theminimum Variance amongst all Unbiased Estimators . We call it theminimumvariance Unbiased estimator(MVUE) of .Sufficiency is a powerful property in finding Unbiased , Minimum Variance estima-tors. IfT(Y)is an Unbiased estimator of andSis a statistic sufficient for ,then there is a function ofSthat is also an Unbiased estimator of and has nolarger Variance than the Variance ofT(Y). The following theorem formalizes (Y1, Y2, .. , Yn)Tbe a random sample,S= (S1, .. , Sp)Tbe jointlysufficient statistics for = ( 1, .. , p)TandT(Y)(which isnota function ofS) be an Unbiased estimator of =g( ). Then,U=E(T|S)is a statistic suchthat(a)E(U) = , so thatUis an Unbiased estimator of , and(b)var(U)<var(T).

2 , we note thatUis a statistic. Indeed, sinceSare jointly sufficient for , the conditional distributionY|Sdoes not depend on the parameters and so theconditional distribution of a functionT(Y)givenS,T|S, does not depend on either. Thus,U= E(T|S)is a function of the random sample only, not a functionof , therefore it is a , we will use the known facts about the conditional expectation and variancegiven Exercise SinceTis an Unbiased estimator of , we haveE(U) = E[E(T|S)] = E(T) = .SoUis also an Unbiased estimator of , which proves (a). Finally, we getvar(T) = var[E(T|S)] + E[var(T|S)]= var(U) + E[var(T|S)].However, sinceTis not a function ofSwe havevar(T|S)>0, thus, it followsthatE[var(T|S)]>0, and hence (b) is proved. 88 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCEIt means that, if we have an Unbiased estimator,T, of , which is not a functionof the sufficient statistics, we can always find an Unbiased estimator which hassmaller Variance , namelyU=E(T|S1.)

3 , Sp)which is a function ofS. Wethus have the following must be functions of sufficient thatY1, .. , Ynare independentPoisson( )random vari-ables. ThenT=Yi, for anyi= 1, .. , n, is an Unbiased estimator of . Also,S= ni=1 Yiis a sufficient statistic for andTis not a function , a better Unbiased estimator is given by any ofE(Y1| ni=1Yi), .. ,E(Yn| ni=1Yi).Now, sinceE(Y1| ni=1Yi) =..= E(Yn| ni=1Yi)andE(Y1|n i=1Yi)+E(Y2|n i=1Yi)+..+E(Yn|n i=1Yi) = E[n i=1Yi|n i=1Yi]=n i=1Yi,we havenE(Y1|n i=1Yi) =n i= E(Y1| ni=1Yi) =Yis a better Unbiased estimator of thanY1orthan any ofYi. In fact,Yis a MVUE of as its Variance is equal to the Cramer-Rao Lower Bound for . (See Example ) Complete sufficient statisticsT(Y)is a function of random sampleY= (Y1, Y2, .. , Yn)and so it is a randomvariable as well.

4 Hence, we may ask about the distribution ofT(Y). For example,assume that 2is known equal to 2o. Then, to make inference about we may reduce the random sample to its mean. We know thatT(Y) =Y N( , 2on)ifYi iidN( , 2o)and we may writefT(t; | 2o) =1 2 oen(t )2/2 to this data reduction we can make inference about based on the distri-bution ofYonly rather than on the multivariate distribution of the whole POINT minimal sufficient statistic reduces data maximally whileretaining all the infor-mation about the parameter . We would also like such a statistic to be indepen-dent of any so calledancillaryfunctions of the random sample whose distributionsdo not depend on the parameter of interest. Such an independent statistic is , we introduce a notion of a complete family of family of distributionsP={P : }defined on a commonspaceYis calledcompleteif for any real measurable functionh(Y)E[h(Y)] = 0implies that P (h(Y) = 0)= 1f or all.

5 Note:P (h(Y) = 0)= 1can also be written asP (h(Y)6= 0)= 0, whichmeans that functionh(Y)may have non-zero values only on a setB YsuchthatP(Y B) = 0. Then we say thath(Y) = 0almost surely statisticT(Y)is calledcompletefor the familyP={P : }onYif the family of probability distributionsPT={P ,T: }iscomplete for all , that is,E[h(T)] = 0implies thatP{h(T) = 0}= 1. (Y1, .. , Yn)be a random sample from a family ofBernoulli(p)distributions for0< p <1. We will show thatT(Y) = ni=1 Yiisa complete sufficient statistic pmf of eachYiisP(Yi=yi) =pyi(1 p)1 yiand the joint pmfforYcan be factorized as follows:P(Y=y) =n i=1pyi(1 p)1 yi=p ni=1yi(1 p)n ni=1yi 1 Hence,T(Y) = ni=1 Yiis a sufficient statistic 2. ELEMENTS OF STATISTICAL INFERENCEC ompletenessNow, we know that a sum of independent Bernoulli rvs has a Bi-nomial distribution, ,T Bin(n, p)for0< p <1, t= 0,1.

6 , (T)be such thatE[h(T)] = 0. Then0 = E[h(T)]=n t=0h(t)nCtpt(1 p)n t= (1 p)nn t=0h(t)nCt(p1 p) factor(1 p)n6= 0for anyp (0,1). Thus, it must be that0 =n t=0h(t)nCt(p1 p)t=n t=0h(t)nCtrtfor allr=p1 p>0. The last expression is a polynomial of the polynomial to be zero for allrthe coefficientsh(t)nCtmust all bezero. It means thath(t) = 0for allt {0,1, .. , n}. SinceP(T=t, t {0,1, .. , n}) = 1it means thatP{h(T) = 0}= 1for allp. Hence,Tis acomplete statistic forp. The following theorem gives a connection between complete and minimal suffi-cient statistics:Theorem (Y)is a complete sufficient statistic for a family of distributionswith parameter , thenT(Y)is a minimal sufficient statistic for the family. thatY1, Y2, .. , Ynis a random sample from aPoisson( )distribution. Show thatT(Y) = ni=1 Yiis a complete sufficient statistic for.

7 The following Theorem establishes the Minimum Variance property of completesufficient POINT ESTIMATION91 Theorem e (Y1, Y2, .. , Yn)Tbe a random sample. IfS(Y)is a jointly completesufficient statistic andT(Y)is an Unbiased estimator for =g( )thenU= E[T|S]is, with probability 1, a unique MVUE of . , to prove thatUis aMV UEofg( ), we show that whatever unbi-ased estimatorT(Y)we take we obtain the sameE[T|S], , the sameU. Then,by Rao-Blackwell Theorem, condition (b),Umust be MVUE ofg( ).Suppose thatT(Y)andT (Y)are any two Unbiased Estimators ofg( ). LetU= E[T|S]U = E[T |S].Then we haveE{U U }= E{E[T|S] E[T |S]}= E(T) E(T )=g( ) g( )= , by completeness ofS(Y)we getP[U(S(Y))=U (S(Y))]= 1for all . This proves the first part of the theorem. Now, we will show thatUandT?are two MVUE ofg( ).

8 Then ifT?is a function of thesufficient statisticsS(Y)then, as shown above, it must be equal toU. IfT?isnot a function ofS(Y)thenvar(U)<var(T?), henceT?cannot be a ,Uis a unique MVUE ofg( ). Note: Lehmann-Scheff e Theorem may be used to construct MVUE ofg( )bytwo methods. Both are based on complete sufficient statisticsS(Y). Method 1: If we can find a function ofS=S(Y), sayU(S)such thatE[U(S)] =g( )thenU(S)is a unique MVUE ofg( ).92 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCE Method 2: If we can find any Unbiased estimatorT=T(Y)ofg( ), thenU(S) = E[T|S]is a unique MVUE ofg( ). 1. LetYi iidBernoulli(p),i= 1, .. , n. Earlier weshowed that ni=1 Yiis a complete sufficient statistic forp. Denote it byS(Y).Y=1n ni=1Yi=1nS(Y)is an Unbiased estimator ofp, hence, as a function of acomplete sufficient statistic, it is the unique MVUE , letg(p) = var(Y) =p(1 p).

9 The sample variance1n 1n i=1(Yi Y)2is an Unbiased estimator ofg(p). It is in fact a function of the complete sufficientstatisticS(Y) = ni=1Yi. Hence, it is the unique MVUE ofg(p) =p(1 p). thatY= (Y1, .. , Yn)Tis a random sample from aPoisson( )distribution. Find a MVUE of = Exponential familiesThere is a class of distributions, including the normal, Poisson, binomial, gamma,chi-squared, exponential and others for which complete sufficient statistics family of probability distributionsP={P : }is calledexponentialif for every distribution belonging to the family, its pdf (pmf) can bewritten in the formf(y; ) =h(y) exp{p j=1aj( )bj(y) +c( )}. Note: For the one-parameter exponential family, this reduces tof(y; ) =h(y) exp{a( )b(y) +c( )}. POINT thatY Bin(m, p), wheremis known.

10 Then we maywriteP(Y=y;p) =(my)py(1 p)m y=(my)exp{ylogp+ (m y) log(1 p)}=(my)exp{ylogp ylog(1 p) +mlog(1 p)}=(my)exp{ylog(p1 p)+mlog(1 p)}.Thus, we havea(p) = log{p/(1 p)},b(y) =y,c(p) =mlog(1 p)andh(y) =(my). Hence,P={P(Y=y;p) :p (0,1)}is an exponential familyof distributions. thatP={P(Y=y; ) = ye y!I{0,1,2,..}: >0}is anexponential family of thatY N( , 2). Then we may writef(y; , 2) =1 2 2e (y )22 2= exp{ 12log(2 2) (y )22 2}= exp{ y22 2+ y 2 22 2 12log(2 2)}= exp{ 2y 12 2y2 22 2 log 12log(2 )}.=1 2 exp{ 2y 12 2y2 22 2 log }.Thus, we havea1( , 2) = / 2,b1(y) =y,a2( , 2) = 1/(2 2),b2(y) =y2,c( , 2) = 2/(2 2) log andh(y) =1 2 . Hence, the family of normaldistributions,P={f(y; , 2) : R, 2 R+}, is a family of exponentialdistributions. 94 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCEL emma (Y1.)