Bootstrap confidence intervals: when, which, what? A ...

1 STATISTICS IN 2000;19:1141}1164 Bootstrap con"dence intervals: when , which , what ?A practical guide for medical statisticiansJames Carpenter1,*and John Bithell21 Medical Statistics Unit, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, of Statistics, 1 South Parks Road, Oxford, OX1 3TG, the early 1980s, a bewildering array of methods for constructing Bootstrap con"dence intervals havebeen proposed. In this article, we address the following questions. First, when should Bootstrap con"denceintervals be used. Secondly, which method should be chosen, and thirdly, how should it be implemented. Inorder to do this, we review the common algorithms for resampling and methods for constructing bootstrapcon"dence intervals, together with some less well known ones, highlighting their strengths and then present a simulation study, a#ow chart for choosing an appropriate method and a survival analysisexample.

2 Copyright(2000 John Wiley & Sons, INTRODUCTION: confidence INTERVALS AND COVERAGE ERRORAn accurate estimate of the uncertainty associated with parameter estimates is important to avoidmisleading inference. This uncertainty is usually summarized by a con"dence interval or region, which is claimed to include the true parameter value with a speci"ed probability. In this paper weshall restrict ourselves to con"dence intervals. We begin with an example which illustrates whywe might want Bootstrap con"dence Example: Remission from acute myelogenous leukaemiaEmburyet al. [1] conducted a clinical trial to evaluate the e$cacy of maintenance chemotherapyfor acute myelogenous leukaemia. Twenty-three patients who were in remission after treatmentwith chemotherapy were randomized into two groups.)

3 The"rst group continued to receivemaintenance chemotherapy, while the second did not. The objective of the trial was to examinewhether maintenance chemotherapy prolonged the time until relapse. The preliminary results ofthe study are shown in Table I. We wish to test the hypothesis that maintenance chemotheraphydoes not delay relapse by constructing a con"dence interval for the treatment e!ect.*Correspondence to: James Carpenter, Medical Statistics Unit, London School of Hygiene and Tropical Medicine,Keppel Street, London WC1E 7HT, January 1998 Copyright(2000 John Wiley & Sons, August 1999 Table I. Remission times (weeks) forpatients with acute myelogenous leukae-mia; group 1 with maintenance chemo-therapy; group 2 none. An entry suchas'13 means that the only informationavailable is that at 13 weeks the patientwas still in 1 Treat 295135'138188121223'1631233427'45304833' 1614345 Figure 1.)

4 Plot of non-parametric estimate of cumulative hazard function for the data in Table I, on a log-logscale. The upper line corresponds to treatment group 1 shows a log-log plot of the cumulative hazard function, and suggests that a propor-tional hazards modelh(t)"h0(t) exp(bx)1142J. CARPENTER AND J. BITHELLC opyright(2000 John Wiley & Sons, 2000;19:1141}1164will be adequate, whereh(t) is the hazard at timet,h0()) is the baseline hazard andxis anindicator covariate for treatment 2. Fitting this model givesbK" with standard errorp(" A standard 95 per cent normal approximation con"dence interval forbis thereforebK$ ] , that is, (! , ).However, the accuracy of this interval depends on the asymptotic normality ofbKand thisassumption is questionable with so few observations.)

5 Accordingly, we may want to constructa con"dence interval that does not depend on this assumption. Bootstrapping provides a ready,reliable way to do this. The principal questions arewhichbootstrap con"dence interval methodshould be chosen andwhatshould be done to implement it. These are the questions this articleseeks to answer in a practical context, by reviewing the methods available, highlighting theirmotivation, strengths and weaknesses. After doing this, we return to this example in Section begin by de"ningcoverage error, which is a key concept in comparing Bootstrap con"denceinterval methods. Suppose (!R,hU) is, for example, a normal approximation con"dence inter-val, with nominal coverage 100(1!a) per cent (atypically ). Then it will often have acoverageerrorso thatP(h(hU)"(1!))

6 A)#Cfor some unknown constantC, where typicallyCP0asn, the sample size, con"dence intervals aim to reduce this coverage error by using simulation to avoidthe assumptions inherent in classical procedures. While they are often calculated for small datasets (for example, to check on the assumption of asymptotic normality), they are equallyapplicable to large data sets and complex models; see for example Carpenter [2] and LePage andBillard [3].Thus, Bootstrap con"dence intervals will at the least validate the assumptions necessary toconstruct classical intervals, while they may further avoid misleading inferences being drawn. Theway they go about this, and the extent on which they succeed, are described in Sections 2 and3 and illustrated in Sections 5 and 7.

7 First, however, we describe thebootstrap principleand many statistical problems we seek information about the value of a population parameterhby drawing a random sampleYfrom that population and constructing an estimatehK(Y) of thevalue ofhfrom that sample. The Bootstrap principle is to obtain information about therelationship betweenhand the random variablehK(Y) by looking at the relationship betweenhK(y0"4) andhK(Y*), whereY*is a resample characterized by the sampley0" *can either beconstructed by sampling with replacement from the data vectory0"4, the so-callednon-parametricbootstrap, or by sampling from the distribution function parameterized byhK(y0"4), the we discuss the various methods for Bootstrap con"dence interval construction, we givealgorithms for non-parametric and parametric simulation, and illustrate these in a regressioncontext, where the Bootstrap is frequently RESAMPLING PLANSHere we give algorithms for non-parametric and semi-parametric resampling plans, and illustratethem with a linear model example.

8 We"rst describe this confidence INTERVALS1143 Copyright(2000 John Wiley & Sons, 2000;19:1141}1164 Table II. Weights of 14 babies at birth and 70}100 days. From Armitageand Berry (Reference [4], p. 148).Case numberBirthweight (oz)Weight at 70}100 days (oz)172121211218331111844107184511918169 2161712622288017498117810841801111514812 1181681312818914128192 Figure 2. Plot of data in Table II, with least squares regression line 70}100 day weight"a#b]birth-weight. Estimates (standard errors) arebK" ( ),a(" ( ).Table II gives the weight at birth and 70}100 days of 14 babies, from Armitage and Berry(Reference [4], p. 148). The data are plotted in Figure 2, together with the least squares regressionline. Parameter estimates are given in the caption. It appears that there is a borderline associationbetween birthweight and weight at 70}100 days.))

9 However, the data set is small, and we may wishto con"rm our conclusions by constructing a Bootstrap con"dence interval for the slope. To dothis, we"rst need to construct Bootstrap versionsbK*,ofb. We now outline how to do thisnon-parametrically and CARPENTER AND J. BITHELLC opyright(2000 John Wiley & Sons, 2000;19:1141} Non-parametric resamplingNon-parametric resampling makes no assumptions concerning the distribution of, or model for,the data. Our data is assumed to be a vectory0"4ofnindependent observations, and we areinterested in a con"dence interval forhK(y0"4). The general algorithm for a non-parametricbootstrap is as follows:1. Samplenobservations randomly with replacement fromy0"4to obtain a Bootstrap data set,denotedY*.2. Calculate the Bootstrap version of the statistic of interest,hK*"hK(Y*).)

10 3. Repeat steps 1 and 2 a large number of times, sayB, to obtain an estimate of the discuss the value ofBappropriate for con"dence intervals in Section the context of the birthweight data in Table II, each&observation'in the original data setconsists of a pair, or case, (x,y). For example, the"rst case is (72, 121). The algorithm thenproceeds as follows:1. Samplencases randomly with replacement to obtain a Bootstrap data set. Thus, a typicalbootstrap data set might select the following Fit the linear model to the Bootstrap data and obtain the Bootstrap slope,bK*. For thespeci"c Bootstrap data set in step 1,bK*" Repeat steps 1 and 2 a large number, sayB, of times to obtain an estimate of the Bootstrap slopesbK*1,2,bK*B, can then be used to form a non-parametric Bootstrap con"-dence interval forbas described in Section Parametric resamplingIn parametric resampling we assume that a parametric model for the data, FY(y.)