Sample size and power calculations - Columbia …

1 CHAPTER 20 Sample size and power Choices in the design of data collectionMultilevel modeling is typically motivated by features in existing data or the objectof study for example, voters classified by demography and geography, students inschools, multiple measurements on individuals, and so on. Consider all the examplesin Part 2 of this book. In some settings, however, multileveldata structures ariseby choice from the data collection process. We briefly discuss some of these sampling or cluster samplingIn a Sample survey, data are collected on a set of units in order to learn about a largerpopulation. In unit sampling, the units are selected directly from the population. Incluster sampling, the population is divided into clusters:first a Sample of clustersis selected, then data are collected from each of the sampling, complete information is collected within each sam-pled cluster.

2 For example, a set of classrooms is selected atrandom from a largerpopulation, and then all the students within each sampled classroom are inter-viewed. Intwo-stagecluster sampling, a Sample is performed within each sampledcluster. For example, a set of classrooms is selected, and then a random Sample often students within each classroom is selected and interviewed. More complicatedsampling designs are possible along these lines, includingadaptive designs, strati-fied cluster sampling, sampling with probability proportional to size , and variouscombinations and elaborations of studies or experiments with unit-level or group-level treatmentsTreatments can be applied (or can be conceptualized as beingapplied in the caseof a purely observational study) at individual or group levels; for example: In a medical study, different treatments might be applied to different patients,with patients clustered within hospitals that could be associated with varyingintercepts or slopes.

3 As discussed in Section , the Electric Company television show was viewedby classes, not individual students. As discussed in Section , child support enforcement policies are set by statesand cities, not individuals. In the radon study described in Chapter 12, we can compare houses with andwithout basements within a county, but we can only study uranium as it variesbetween present a longer list of such designs in the context of experiments in size AND power CALCULATIONST ypically, coefficients for factors measured at the individual level can be esti-mated more accurately than for group-level factors becausethere will be more indi-viduals than groups; so 1/ nis more effective than 1/ Jat reducing the Sample size of a study can be increased in several ways: Gathering more data of the sort already in the study, Including more observations either in a nonclustered setting, as new observationsin existing clusters, or new observations in new clusters Finding other studies performed under comparable (but not identical) conditions(so new observations in effect are like observations from a new group ).

4 Finding other studies on related phenomena (again new observations from adifferent group ).For example, in the study of teenage smoking in Section ,these four optionscould be: (a) surveying more Australian adolescents about their smoking behav-ior, (b) taking more frequent measurements (for example, asking about smokingbehavior every three months instead of every six months), (c) performing a sim-ilar survey in other cities or countries, or (d) performing similar studies of otherunhealthy first option is most straightforward increasingndecreases standard errorsin proportion to 1/ n. The others involve various sorts of multilevel models andare made more effective by collecting appropriate predictors at the individual andgroup levels.

5 (As discussed in Section , the more that the variation is explainedby external predictors, the more effective the partial pooling will be.) A challengeof multilevel design is to assess the effectiveness of these various strategies forincreasing Sample size . Finding data from other studies is often more feasible thanincreasingnin an existing study, but then it is important to either find other studiesthat are similar, or to be able to model these size , design, and interactionsSample size is never large enough. Asnincreases, we estimate more interactions,which typically are smaller and have relatively larger standard errors than maineffects (for example, see the fitted regression on page 63 of log earnings on sex,standardized height, and their interaction).

6 Estimating interactions is similar tocomparing coefficients estimated from subsets of the data (for example, the co-efficient for height among men, compared to the coefficient among women), thusreducing power because the Sample size for each subset is halved, and also thedifferences themselves may be small. As more data are included in an analysis, itbecomes possible to estimate these interactions (or, usingmultilevel modeling, toinclude them and partially pool them as appropriate), so this is not a problem. Weare just emphasizing that, just as you never have enough money, because perceivedneeds increase with resources, your inferential needs willincrease with your power calculations FOR Classical power calculations : general principles, asillustrated byestimates of proportionsQuestions of data collection can typically be expressed in terms of estimates andstandard errors for quantities of interest.

7 This chapter follows the usual focus onestimating population averages, proportions, and comparisons in Sample surveys;or estimating treatment effects in experiments and observational studies. However,the general principles apply for other inferential goals such as prediction and datareduction. The paradigmatic problem of power calculation is the estimation of aparameter (for example, a regression coefficient such as would arise in estimatinga difference or treatment effect), with the Sample size determining the sizes and Sample sizesIn designing a study to maximize the power of detecting a statistically significantcomparison, it is generally better, if possible, to double the effect size than todouble the Sample sizen, since standard errors of estimation decrease with thesquare root of the Sample size .

8 This is one reason, for example, why potentialtoxins are tested on animals at many times their exposure levels in humans; seeExercise are designed in several ways to maximize effect size : In drug studies, setting doses as low as ethically possible in the control groupand as high as ethically possible in the experimental group. To the extent possible, choosing individuals that are likely to respond stronglyto the treatment. For example, the Electric Company experiment described inSection was performed on poorly performing classes in each grade, for whichit was felt there was more room for practice, this advice cannot be followed completely. In the social sciences, itcan be difficult to find an intervention withanynoticeable positive effect, let aloneto design one where the effect would be doubled.

9 Also, when treatments in anexperiment are set to extreme values, generalizations to more realistic levels can besuspect; in addition, missing data in the control group may be more of a problemif the control treatment is ineffective. Further, treatmenteffects discovered on asensitive subgroup may not generalize to the entire population. But, on the whole,conclusive effects on a subgroup are generally preferred to inconclusive but moregeneralizable results, and so conditions are usually set upto make effects as largeas calculationsBefore data are collected, it can be useful to estimate the precision of inferencesthat one expects to achieve with a given Sample size , or to estimate the Sample sizerequired to attain a certain precision.

10 This goal is typically set in one of two ways: Specifying the standard error of a parameter or quantity to be estimated, or Specifying the probability that a particular estimate willbe statistically signif-icant, which typically is equivalent to ensuring that its confidence interval willexclude the null either case, the Sample size calculation requires assumptions that typically cannotreally be tested until the data have been collected. Sample size calculations are thusinherently size AND power of p^ (based on n=96)p^possible 95% intervals (based on n=96) of p^ (based on n=196)p^possible 95% intervals (based on n=196) of simple Sample size row: (a) distribution of the Sample proportion pif the true population proportion isp= , based on a Sample size of 96; (b) several possible 95% intervals forpbased ona Sample size of 96.