1 Sufficient statistics

1 1 Sufficient statisticsAstatisticis a functionT=r(X1, X2, , Xn) of the random sampleX1, X2, , are Xn=1nn i=1Xi,(the sample mean)s2= =1n 1n i=1(Xi Xn)2,(the sample variance)T1= max{X1, X2, , Xn}T2= 5(1)The last statistic is a bit strange (it completely igonores the random sample),but it is still a statistic. We say a statisticTis an estimator of a populationparameter ifTis usually close to . The sample mean is an estimator forthe population mean; the sample variance is an estimator for the , there are lots of functions ofX1, X2, , Xnand so lots ofstatistics. When we look for a good estimator, do we really need to considerall of them, or is there a much smaller set of statistics we could consider?Another way to ask the question is if there are a few key functions of therandom sample which will by themselves contain all the information thesample does.

2 For example, suppose we know the sample mean and the samplevariance. Does the random sample contain any more information about thepopulation than this? We should emphasize that we are always assuming ourpopulation is described by a given family of distributions (normal, binomial,gamma or ..) with one or several unknown parameters. The answer tothe above question will depend on what family of distributions we assumedescribes the start with a heuristic definition of asufficient sayTisa sufficient statistic if the statistician who knows the value ofTcan do justas good a job of estimating the unknown parameter as the statistician whoknows the entire random mathematical definition is as follows. A statisticT=r(X1, X2, , Xn)is a sufficient statistic if for eacht, the conditional distribution ofX1, X2, , XngivenT=tand does not depend on.

3 1To motivate the mathematical definition, we consider the following ex-periment. LetT=r(X1, , Xn) be a sufficient statistic. There are twostatisticians; we will call them A and B. Statistician A knows the entire ran-dom sampleX1, , Xn, but statistician B only knows the value ofT, call itt. Since the conditional distribution ofX1, , Xngiven andTdoes notdepend on , statistician B knows this conditional distribution. So he canuse his computer to generate a random sampleX 1, , X nwhich has thisconditional distribution. But then his random sample has the same distri-bution as a random sample drawn from the population (with its unknownvalue of ). So statistician B can use his random sampleX 1, , X nto com-pute whatever statistician A computes using his random sampleX1, , Xn,and he will (on average) do as well as statistician A.

4 Thus the mathematicaldefinition of sufficient statistic implies the heuristic is difficult to use the definition to check if a statistic is sufficient or tofind a sufficient statistic. Luckily, there is a theorem that makes it easy tofind sufficient 1.(Factorization theorem) LetX1, X2, , Xnbe a random sam-ple with joint densityf(x1, x2, , xn| ). A statisticT=r(X1, X2, , Xn)is sufficient if and only if the joint density can be factored as follows:f(x1, x2, , xn| ) =u(x1, x2, , xn)v(r(x1, x2, , xn), )(2)whereuandvare non-negative functions. The functionucan depend onthe full random samplex1, , xn, but not on the unknown parameter .The functionvcan depend on , but can depend on the random sample onlythrough the value ofr(x1, , xn).It is easy to see that iff(t) is a one to one function andTis a sufficientstatistic, thenf(T) is a sufficient statistic.

5 In particular we can multiply asufficient statistic by a nonzero constant and get another sufficient now apply the theorem to some (normal population, unknown mean, known variance)We considera normal population for which the mean is unknown, but the the variance 2is known. The joint density isf(x1, , xn| ) = (2 ) n/2 nexp( 12 2n i=1(xi )2)2= (2 ) n/2 nexp( 12 2n i=1x2i+ 2n i=1xi n 22 2)Since 2is known, we can letu(x1, , xn) = (2 ) n/2 nexp( 12 2n i=1x2i)andv(r(x1, x2, , xn), ) = exp( n 22 2+ 2r(x1, x2, , xn))wherer(x1, x2, , xn) =n i=1xiBy the factorization theorem this shows that ni=1 Xiis a sufficient statis-tic. It follows that the sample mean Xnis also a sufficient (Uniform population)Now suppose theXiare uniformly dis-tributed on [0, ] where is unknown. Then the joint density isf(x1, , xn| ) = n1(xi , i= 1,2, , n)Here 1(E) is an indicator function.

6 It is 1 if the eventEholds, 0 if it doesnot. Nowxi fori= 1,2, , nif and only if max{x1, x2, , xn} .So we havef(x1, , xn| ) = n1(max{x1, x2, , xn} )By the factorization theorem this shows thatT= max{X1, X2, , Xn}is a sufficient about the sample mean? Is it a sufficient statistic in this example?By the factorization theorem it is a sufficient statistic only if we can write1(max{x1, x2, , xn} ) as a function of just the sample mean and . Thisis impossible, so the sample mean is not a sufficient statistic in this (Gamma population, unknown, known)Now suppose the pop-ulation has a gamma distribution and we know but is unknown. Thenthe joint density isf(x1, , xn| ) = n ( )n(n i=1x 1i)exp( n i=1xi)We can writen i=1x 1i= exp(( 1)n i=1ln(xi))By the factorization theorem this shows thatT=n i=1ln(Xi)is a sufficient statistic.

7 Note that exp(T) = ni=1 Xiis also a sufficientstatistic. But the sample mean is not a sufficient we reconsider the example of a normal population, but suppose thatboth and 2are both unknown. Then the sample mean is not a sufficientstatistic. In this case we need to use more than one statistic to get definition (both heuristic and mathematical) of sufficiency extends toseveral statistics in a natural considerkstatisticsTi=ri(X1, X2, , Xn), i= 1,2, , k(3)We sayT1, T2, , Tkare jointly sufficient statistics if the statistician whoknows the values ofT1, T2, , Tkcan do just as good a job of estimatingthe unknown parameter as the statistician who knows the entire randomsample. In this setting typically represents several parameters and thenumber of statistics ,k, is equal to the number of unknown mathematical definition is as follows.

8 The statisticsT1, T2, , Tkare jointly sufficient if for eacht1, t2, , tk, the conditional distribution ofX1, X2, , XngivenTi=tifori= 1,2, , nand does not depend on .Again, we don t have to work with this definition because we have thefollowing theorem:4 Theorem 2.(Factorization theorem) LetX1, X2, , Xnbe a random sam-ple with joint densityf(x1, x2, , xn| ). The statisticsTi=ri(X1, X2, , Xn), i= 1,2, , k(4)are jointly sufficient if and only if the joint density can be factored as follows:f(x1, x2, , xn| ) =u(x1, x2, , xn)v(r1(x1, , xn), r2(x1, , xn), , rk(x1, , xn), )whereuandvare non-negative functions. The functionucan depend onthe full random samplex1, , xn, but not on the unknown parameters .The functionvcan depend on , but can depend on the random sample onlythrough the values ofri(x1, , xn),i= 1,2, , that for a single statistic we can apply a one to one function tothe statistic and get another sufficient statistic.

9 This generalizes as (t1, t2, , tk) be a function whose values are inRkand which is oneto one. Letgi(t1, , tk),i= 1,2, , kbe the component functions ifT1, T2, , Tkare jointly sufficient, thengi(T1, T2, , Tk) are now revisit two examples. First consider a normal population withunknown mean and variance. Our previous equations show thatT1=n i=1Xi, T2=n i=1X2iare jointly sufficient statistics . Another set of jointly sufficent statistics isthe sample mean and sample variance. (What isg(t1, t2) ?)Now consider a population with the gamma distribution with both and unknown. ThenT1=n i=1Xi, T2=n i=1ln(Xi)are jointly sufficient Exercises1. Gamma, both parameters unknown, show sum and product form a suffi-cient2. Uniform on [ 1, 2]. Find two statistics that are Poisson or geometric - find sufficient Beta ?

10 6