Bootstrap confidence intervals: when, which, …

1 STATISTICS IN 2000;19:1141}1164 Bootstrap con"dence intervals: when , which , what?A practical guide for medical statisticiansJames Carpenter1,*and John Bithell21 Medical Statistics Unit, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, of Statistics, 1 South Parks Road, Oxford, OX1 3TG, the early 1980s, a bewildering array of methods for constructing Bootstrap con"dence intervals havebeen proposed. In this article, we address the following questions. First, when should Bootstrap con"denceintervals be used.

2 Secondly, which method should be chosen, and thirdly, how should it be implemented. Inorder to do this, we review the common algorithms for resampling and methods for constructing bootstrapcon"dence intervals, together with some less well known ones, highlighting their strengths and then present a simulation study, a#ow chart for choosing an appropriate method and a survival analysisexample. Copyright(2000 John Wiley & Sons, INTRODUCTION: confidence INTERVALS AND COVERAGE ERRORAn accurate estimate of the uncertainty associated with parameter estimates is important to avoidmisleading inference.)

3 This uncertainty is usually summarized by a con"dence interval or region, which is claimed to include the true parameter value with a speci"ed probability. In this paper weshall restrict ourselves to con"dence intervals. We begin with an example which illustrates whywe might want Bootstrap con"dence Example: Remission from acute myelogenous leukaemiaEmburyet al. [1] conducted a clinical trial to evaluate the e$cacy of maintenance chemotherapyfor acute myelogenous leukaemia. Twenty-three patients who were in remission after treatmentwith chemotherapy were randomized into two groups.

4 The"rst group continued to receivemaintenance chemotherapy, while the second did not. The objective of the trial was to examinewhether maintenance chemotherapy prolonged the time until relapse. The preliminary results ofthe study are shown in Table I. We wish to test the hypothesis that maintenance chemotheraphydoes not delay relapse by constructing a con"dence interval for the treatment e!ect.*Correspondence to: James Carpenter, Medical Statistics Unit, London School of Hygiene and Tropical Medicine,Keppel Street, London WC1E 7HT, January 1998 Copyright(2000 John Wiley & Sons, August 1999 Table I.)

5 Remission times (weeks) forpatients with acute myelogenous leukae-mia; group 1 with maintenance chemo-therapy; group 2 none. An entry suchas'13 means that the only informationavailable is that at 13 weeks the patientwas still in 1 Treat 295135'138188121223'1631233427'45304833' 1614345 Figure 1. Plot of non-parametric estimate of cumulative hazard function for the data in Table I, on a log-logscale. The upper line corresponds to treatment group 1 shows a log-log plot of the cumulative hazard function, and suggests that a propor-tional hazards modelh(t)"h0(t) exp(bx)1142J.

6 CARPENTER AND J. BITHELLC opyright(2000 John Wiley & Sons, 2000;19:1141}1164will be adequate, whereh(t) is the hazard at timet,h0()) is the baseline hazard andxis anindicator covariate for treatment 2. Fitting this model givesbK" with standard errorp(" A standard 95 per cent normal approximation con"dence interval forbis thereforebK$ ] , that is, (! , ).However, the accuracy of this interval depends on the asymptotic normality ofbKand thisassumption is questionable with so few observations. Accordingly, we may want to constructa con"dence interval that does not depend on this assumption.)

7 Bootstrapping provides a ready,reliable way to do this. The principal questions arewhichbootstrap con"dence interval methodshould be chosen andwhatshould be done to implement it. These are the questions this articleseeks to answer in a practical context, by reviewing the methods available, highlighting theirmotivation, strengths and weaknesses. After doing this, we return to this example in Section begin by de"ningcoverage error, which is a key concept in comparing Bootstrap con"denceinterval methods. Suppose (!R,hU) is, for example, a normal approximation con"dence inter-val, with nominal coverage 100(1!)

8 A) per cent (atypically ). Then it will often have acoverageerrorso thatP(h(hU)"(1!a)#Cfor some unknown constantC, where typicallyCP0asn, the sample size, con"dence intervals aim to reduce this coverage error by using simulation to avoidthe assumptions inherent in classical procedures. While they are often calculated for small datasets (for example, to check on the assumption of asymptotic normality), they are equallyapplicable to large data sets and complex models; see for example Carpenter [2] and LePage andBillard [3].

9 Thus, Bootstrap con"dence intervals will at the least validate the assumptions necessary toconstruct classical intervals, while they may further avoid misleading inferences being drawn. Theway they go about this, and the extent on which they succeed, are described in Sections 2 and3 and illustrated in Sections 5 and 7. First, however, we describe thebootstrap principleand many statistical problems we seek information about the value of a population parameterhby drawing a random sampleYfrom that population and constructing an estimatehK(Y) of thevalue ofhfrom that sample.

10 The Bootstrap principle is to obtain information about therelationship betweenhand the random variablehK(Y) by looking at the relationship betweenhK(y0"4) andhK(Y*), whereY*is a resample characterized by the sampley0" *can either beconstructed by sampling with replacement from the data vectory0"4, the so-callednon-parametricbootstrap, or by sampling from the distribution function parameterized byhK(y0"4), the we discuss the various methods for Bootstrap con"dence interval construction, we givealgorithms for non-parametric and parametric simulation, and illustrate these in a regressioncontext, where the Bootstrap is frequently resampling PLANSHere we give algorithms for non-parametric and semi-parametric resampling plans, and illustratethem with a linear model example.