Small studies: strengths and limitations

1 EDITORIALS mall studies: strengths and limitationsA. HackshawAlarge number of clinical research studies are con-ducted, including audits of patient data, observationalstudies, clinical trials and those based on laboratoryanalyses. While Small studies can be published over a shorttime-frame, there needs to be a balance between those that canbe performed quickly and those that should be based on moresubjects and hence may take several years to complete. Thepresent article provides an overview of the main consider-ations associated with Small Small IS Small ?The definition of Small depends on the main study simply describing the characteristics of a single group ofsubjects, for example the prevalence of smoking, the larger thestudy the more reliable the results.

2 The main results shouldhave 95% confidence intervals (CI), and the width of thesedepend directly on the sample size: large studies producenarrow intervals and, therefore, more precise results. A studyof 20 subjects, for example, is likely to be too Small for mostinvestigations. For example, imagine that the proportion ofsmokers among a particular group of 20 individuals is 25%.The associated 95% CI is 9 49. This means that the trueprevalence in these subjects generally is anywhere between alow or high value, which is not a useful comparing characteristics between two or more groupsof subjects ( risk factors or treatments fordisease), the size of the study depends on the magnitude ofthe expected effect size, which is usually quantified by arelative risk, odds ratio, absolute risk difference, hazard ratio,or difference between two means or medians.

3 The smaller thetrue-effect size, the larger the study needs to be [1, 2]. This isbecause it is more difficult to distinguish between a real effectand random variation. Consider mortality as the end-point in atrial comparing drug A and a placebo with 100 subjects pergroup. If the 1-yr death rate is 15% for drug A and 20% for theplacebo, the risk difference is 5%, but this represents only fivefewer deaths associated with drug A. It is not easy todetermine whether this difference is due to the action of thenew drug or simply chance. There could just happen to be fivefewer deaths in one group. However, if the death rates were 5versus40%, this represents 35 fewer deaths among 100 subjectsreceiving drug A, which are unlikely to all be due to , a trial of 100 patients per arm is too Small if theexpected difference is 5%, but large enough if the expecteddifference is 35%.

4 Figure 1 illustrates how study size influencesthe conclusions that can be with a Small number of subjects can be quick toconduct with regard to enrolling patients, reviewing patientrecords, performing biochemical analyses or asking subjects tocomplete study questionnaires. Therefore, an obvious strengthis that the research question can be addressed in a relativelyshort space of time. Furthermore, Small studies often only needto be conducted over a few centres. Obtaining ethical andinstitutional approval is easier in Small studies compared withlarge multicentre studies. This is particularly true for inter-national is often better to test a new research hypothesis in a smallnumber of subjects first.

5 This avoids spending too manyresources, , time and financial costs, on finding anassociation between a factor and a disorder when there reallyis no effect. However, if an association is found it is importantto make it clear in the conclusions that it was from ahypothesis-generating study and a larger confirmatory studyis studies can also make use of surrogate markers whenexamining associations, factor that can be used instead ofa true outcome measure, but it may not have an obviousimpact that subjects are able to identify. For example, in lungcancer, the true end-point in a clinical trial of a newintervention is overall survival: time until death from anycause.

6 Death is clearly clinically meaningful to patients andclinicians, thus if the intervention increases survival time thisshould provide sufficient justification to change practice. Asurrogate marker is tumour response, or partialremission of the cancer. Surrogate end-points are oftenassociated with more events, which are observed relativelysoon after the intervention is administered; therefore, subjectsmay not require a long follow-up period. Both of thesecharacteristics allow a smaller study to be conducted in ashort space of time. Observing no change in the surrogatemarker usually indicates there is unlikely to be an effect on thetrue end-point, thus avoiding an unnecessary large main problem with Small studies is interpretation ofresults, in particular confidence intervals and p-values (fig.)

7 1).When conducting a research study, the data is used to estimatethe true effect using the observed estimate and 95% confidenceUniversity College London, Cancer Research UK & UCL Cancer Trials Centre, University CollegeLondon, London, OF INTEREST: None : A. Hackshaw, University College London, Cancer Research UK & UCL CancerTrials Centre, University College London, 90 Tottenham Court Road, London W1T 4TJ, UK. Fax: 442076799899. E-mail: Respir J 2008; 32: 1141 1143 DOI: ERS Journals Ltd 2008cEUROPEAN RESPIRATORY JOURNALVOLUME 32 NUMBER 51141interval. Consider hypothetical clinical trials evaluating fournew diets for reducing body weight (table 1).

8 The results fordiet A are clear: they are clinically important (the weight loss islarge) and highly statistically significant (the p-value is verysmall, indicating that the observed weight loss of 7 kg isunlikely to be due to chance). The true mean weight lossassociated with the new diet is estimated to be 7 kg, but thereis 95% certainty that the true value lies somewhere between kg. Ideally all intervals should be as narrow as this, butusually only large studies can produce such precise results. Indiets B and D, the confidence intervals are also narrow, but allaround a Small and clinically unimportant effect so one can befairly confident that these diets are not worthwhile.

9 Thestatistically significant result for diet B is simply due toperforming a very large study, but it would not justify usingthe new most difficult results to interpret are those for diet the confidence interval includes zero, most of therange is below zero and the p-value is just above theconventional cut-off value of This is likely to be due tothe study not being large enough. The data must be interpretedcarefully. The lack of statistical significance does not meanthere is no effect [3], because the true mean weight loss couldbe 3 kg, or even as large as kg.

10 It is better to say there issome evidence of an effect, but the result has just missedstatistical significance , or there is a suggestion of an effect .There needs to be a careful balance between not dismissingoutright what could be a real effect and also not making undueclaims about the major limitation of Small studies is that they canproduce false-positive results, or they over-estimate themagnitude of an association. Table 2 illustrates this limitationusing trials that have evaluated thalidomide in treating lungcancer [4, 5]. After the smaller studies were reported, there wasmuch hope for thalidomide, particularly because it is adminis-tered orally.