Lecture 12 -- Another way to find the Best Estimator

1 1 Lecture 12 -- Another way to find the best Estimator 1. (Regular) Exponential Family The density function of a regular exponential family is: ( ) ( ) ( ) [ ( ) ( ) ] ( ) Example. Poisson( ) ( ) ( ) [ ( ) ] Example. Normal. ( ) ( ) (both unknown). ) ( ) [ ( ) ] [ ( )] [ ] [ ( )] [ ] [ ( ) ( )] 2. Theorem (Exponential family & sufficient Statistic). Let be a random sample from the regular exponential family. Then ( ) ( ( ) ( ) ) is sufficient for ( ) 2 Example.

2 Poisson( ) Let be a random sample from Poisson( ) Then ( ) is sufficient for Example. Normal. ( ) ( ) (both unknown). Let be a random sample from ( ) Then ( ) ( ) is sufficient for ( ) Exercise. Apply the general exponential family result to all the standard families discussed above such as binomial, Poisson, normal, exponential, gamma. A Non-Exponential Family Example. Discrete uniform. ( ) is a positive integer. Another Non-exponential Example. iid ( ) ( ) 3 Universal Cases.

3 Are iid with density . The original data are always sufficient for . (They are trivial statistics, since they do not lead any data reduction) Order statistics ( ( ) ( )) are always sufficient for . ( The dimension of order statistics is , the same as the dimension of the data. Still this is a nontrivial reduction as ! different values of data corresponds to one value of . ) 3. Theorem (Rao-Blackwell) Let be a random sample from the population with pdf ( ). Let ( ) be a sufficient statistic for , and ( ) be any unbiased Estimator of.

4 Let ( ) [ ( ) ], then (1) ( ) is an unbiased Estimator of (2) ( ) is a function of T, (3) ( ) ( ) for every , and ( ) ( ) for some unless with probability 1 . Rao-Blackwell theorem tells us that in searching for an unbiased Estimator with the smallest possible variance ( , the best Estimator , also called the uniformly minimum variance unbiased Estimator UMVUE, which is also referred to as simply the MVUE), we can restrict our search to only unbiased functions of the sufficient statistic T(X). 4 Proof: Make use of the following equations: ( ) [ ( )] ( ) [ ( )] [ ( )] Note: The fact that ( ) is a sufficient statistic for will ensure that ( ) is a function of only the sample and in particular, is independent of.

5 5 4. Transformation of Sufficient Statistics 1. If is sufficient for and ( ) a mathematical function of some other statistic, then is also sufficient. 2. If is sufficient for , and ( ) with being one-to-one, then is also sufficient. Remark: When one statistic is a function of the other statistic and vise verse, then they carry exactly the same amount of information. Examples: If is sufficient, so is . If ( ) are sufficient, so is ( ). If is sufficient, so is ( ) is sufficient, and so is ).

6 Examples of non-sufficiency. Ex. iid Poisson( ). is not sufficient. Ex. iid pmf ( ). ( ) is not sufficient. 6 5. Minimal Sufficient Statistics It is seen that different sufficient statistics are possible. Which one is the " best "? Naturally, the one with the maximum reduction. For ( ), is a better sufficient statistic for than ( ) Definition: is a minimal sufficient statistic if, given any other sufficient statistic , there is a function ( ) such that ( ). Equivalently, is minimal sufficient if, given any other sufficient statistic whenever and are two data values such that ( ) ( ), then ( ) ( ).

7 Partition Interpretation for Minimal Sufficient Statistics: Any sufficient statistic introduces a partition on the sample space. The partition of a minimal sufficient statistic is the coarsest. Minimal sufficient statistic has the smallest dimension among possible sufficient statistics. Often the dimension is equal to the number of free parameters (exceptions do exist). Theorem (How to check minimal sufficiency). A statistic T is minimal sufficient if the following property holds: For any two sample points x and y ( ) ( ) does not depend on ( ( ) ( ) is a constant function of ) if and only if ( ) ( ) 7 6.

8 Exponential Families & Minimal Sufficient Statistic: For a random sample from the regular exponential family with probability density ( ) ( ) ( ) [ ( ) ( ) ], where is k dimensional, the statistic ( ) ( ( ) ( ) ) is minimal sufficient for . Example. Poisson( ) Let be a random sample from Poisson( ) Then ( ) is minimal sufficient for Example. Normal. ( ) ( ) (both unknown). Let be a random sample from ( ) Then ( ) ( ) Is minimal sufficient for ( ) Remarks: Minimal sufficient statistic is not unique.

9 Any two are in one-to-one correspondence, so are equivalent. 8 7. Complete Statistics Let a parametric family ( ) be given. Let be a statistic. Induced family of distributions ( ) . A statistic is complete for the family ( ) or equivalently, the induced family ( ) is called complete, if ( ( )) for all implies that ( ) with probability 1. Example. Poisson( ) Let be a random sample from Poisson( ) Then ( ) is minimal sufficient for Now we show that T is also complete. We know that ( ) ( ) Consider any function ( ).

10 We have [ ( )] ( ) Because setting [ ( )] requires all the coefficient ( ) to be zero, which implies ( ) Example. Let be iid from ( ). Show is a complete statistic. (*Please read our text book for more examples but the following result on the regular exponential family is the most important.) 9 8. Exponential Families & Complete Statistics Theorem. Let be iid observations from the regular exponential family, with the pdf ( ) ( ) ( ) [ ( ) ( ) ], and ( ) Then ( ) ( ( ) ( ) ) is complete if the parameter space ( ( ) ( )) contains an open set in.