SUGI 24: Multiple Comparisons and Multiple Tests …

1 Advances in Multiple Comparisons and Multiple Tests using the SAS SystemPeter H. Westfall, Mail Stop 2101, Texas Tech University, Lubbock TX 79409 Randall D. Tobias, SAS Institute, IntroductionThis article provides a summary of theupcoming book Multiple Comparisons andMultiple Tests using the SAS System, byWestfall et al. (1999) and introduces someof the new methods and capabilities you want to make more than onedecision in a statistically valid way, multipleinferences are involved. There arenumerous alternative solutions for multipleinference problems some are very good,some perform reasonably well, and some areof questionable value. The wide variety ofmethods that are available can make thechoice of technique difficult. In Westfall etal. (1999) we explain the various methods,along with their pitfalls and material in the book is self-contained,requiring only a moderate statisticalbackground.

2 We apply the methods to realdata, giving examples from many areas,including business, medicine, sociology, goal we had in writing this book was tounify the presentation of the diverse multipleinference methods, and to simplify the useof software. The proper choice of amultiple inference procedure depends uponyour inference objectives and data there are procedures in SAS/STAT software for such inferences, such as theGLM, MIXED, and MULTTEST procedures, we realized through the courseof writing the book that several types ofproblems "fall through the cracks," and arenot particularly well accommodated by anyof the existing procedures. To fill this gap,we developed a set of SAS macro languageprograms that implement more general andmore recently developed Multiple inferencetechniques than are available in theprocedures mentioned above.

3 In addition tohandling the usual pairwise comparisonapplications, these macros can be used forproblems as diverse as confidence bands for regressionfunctions (linear, logistic, survivalanalysis,..), simultaneous intervals for log-oddsratios in logistic regression, and closed testing for covariate-adjustedlinear contrasts in multivariate analysisof macros, in conjunction with theexisting facilities for Multiple comparisonsin the SAS system, allow users to carry outmultiple inferences in most applications ofpractical interest. All macros will be madeavailable in the book and on the SAS programs in the book are designed torun in Version 7 of the SAS System. Wetake liberal advantage of such features asvariable and data set names with more thaneight characters, as well as the OutputDelivery System. Therefore, many of ourprograms will not run correctly in releasesprior to Version 7 without suitablemodifications.

4 In particular, two of themore important macros, %SimIntervals and%SimTests, would require extensivemodifications to run in previous Comparisons versusMultiple TestsWhy is there a distinction between multiplecomparisons and Multiple Tests shown in ourtitle? " Multiple Comparisons " usually refersto the comparison of mean values from anexperiment with Multiple groups. ForStatistics, Data Analysis, and Modelingexample, you might compare consumerperceptions of three different advertisingdisplays, labeled A, B, and C, using the datato compare display A with display B, A withC, and B with C. This is the classic" Multiple Comparisons " application, andSAS software has long offered a variety ofmethods for such analyses ( , Tukey smethod for comparing means in PROCGLM). Multiple testing, on the other hand,concerns a broader class of example, a clinical trial designed toassess "efficacy" of a pharmaceuticalcompound might be considered"efficacious" if it reduces fever, or if itspeeds recovery time, or if it reducesheadaches.

5 Here, there are three Tests acomparison of active compound withplacebo for each of the three is an example of " Multiple testing."The distinction between multiplecomparisons and Multiple Tests is that, withmultiple Comparisons , you typicallycompare three of more mean values of thesame measurement, while with multipletesting, you consider aim of our book is to balance thepresentation of Multiple Comparisons withmultiple testing, thereby filling a gap inprevious expositions. There have beenfewer methods proposed and thus littlesoftware developed for analyzing multipletest data, due to difficulties relating to thecovariances among the variables. Forexample, Jason Hsu s excellent book onmultiple Comparisons (Hsu, 1996) does nottreat this problem at all. One aim of ourbook is to address this lack. We givenumerous examples of Multiple testing data,and use the SAS system to solve theproblems.

6 We also give numerous examplesfrom the Multiple Comparisons side, as wellas examples that are a combination of fields of Multiple Comparisons andmultiple testing have tended to consist ofsomething of a hodge-podge of of organizing the book around thesemethods, we ve organized it by the differentmultiple comparison /testing problems youmight want to solve. This makes theinterrelationships between the methodsclearer and makes it easier for you to decidewhich analysis is appropriate. The topicscovered in the book as well as the softwaretools that are used are shown in Table ExamplesOne feature of the book is that all of themethods discussed are illustrated by usingspecific SAS software for some of the mostpowerful methods currently available on realdata from a broad range of following examples give a good idea ofthe range of examples in the book.

7 The firstemploys just the GLM procedure to performa traditional Multiple Comparisons analysis,while the other two demonstrate two of themacros that were developed to implementmore modern - A Simple Balanced ANOVAE xampleThe following data from Ott (1988) compareweight losses for patients on 5 differenttreatment groups. The study is balanced,with 10 subjects per group. Display 1 showshow to analyze the data, using Tukey smethod (Tukey, 1953).Statistics, Data Analysis, and ModelingTable 1 - OutlineTopic/ChapterSoftware ToolsIntroductory Material 1: Multiple Comparisons /Testing Applications 2: General Concepts: Adjusted p-values, p-value plotsPROC MULTTEST, %Rom,%HochBen macrosComparing treatment means 3: Balanced one-way 4: Unbalanced one-way 5: Designs with covariates 6: General functions of means, Confidence Bands 7: Power and sample size 8: Step-down and closure-based testingPROC GLM/MEANS statementPROC GLM/LSMEANS statement%SimIntervals macro%SimIntervals macro%SimPower, %PlotSimPower macrosREGWQ, PROC MULTTEST%Beggab macro, %Simtests macroMore complicated designs 9: Generalizations of one-way methods for standard linear models 10: Heteroscedastic, mixed, and multivariate models 11: Non-normal error distributions 12.

8 Binary Data%SimIntervals, %SimTests macrosPROC MIXED/LSMEANS statement;%SimIntervals, %SimTests macrosPROC MULTTESTPROC MULTTEST, %MultComp macro 13: Bayesian Methods%BayesIntervals, %BayesTestsmacros 14: Further topics Logistic regression Survival analysis Multiple Comparisons with the best%SimIntervals, %SimTests macrosPROC MULTTEST, %SimIntervals,%SimTests macros%MCB macroDisplay 1. Simple Balanced ANOVA: Program and Outputdata wloss; do diet = A , B , C , D , E ; do i = 1 to 10; input wloss @@; output; end; end;datalines; ;proc glm data=wloss; class diet; model wloss=diet; lsmeans diet / pdiff cl adjust=tukey; ods select DiffMat LSMeanDiffCl;run;Statistics, Data Analysis, and ModelingNote the following: Because the data are balanced, theTukey method is exact: the simultaneouscoverage level of the pairwise intervalsis exactly 95% under the usualassumptions.

9 The pairwise p-values matrix showswhich simultaneous intervals excludezero, and at what significance levels,thereby allowing simple determinationsof significance. The program uses a new Version 7 ODSfeature to specify that only the p-valuesand confidence limits for the pairwisecomparisons should be the ODS SELECT statement is new,the application of Tukey s method forbalanced ANOVA has been available in theSAS System for more than 15 years. Next,we consider an application that isconsiderably more complex and shows offsome of the newer - Multiple Tests with MultivariateAnalysis of CovarianceSuppose you have multivariate Multiple -group response data with covariates, whereyou want to perform the Multiple testsassociated with a multivariate analysis ofcovariance, or MANCOVA. A MANCOVA example like this is discussed by Morrison $GMXVWPHQW IRU 0 XOWLSOH &RPSDULVRQV 7 XNH\ /HDVW 6 TXDUHV 0 HDQV IRU HIIHFW GLHW 3U !

10 _W_ IRU + /60 HDQ L /60 HDQ M 'HSHQGHQW 9 DULDEOH ZORVV L M /HDVW 6 TXDUHV 0 HDQV IRU (IIHFW GLHW 'LIIHUHQFH 6 LPXOWDQHRXV %HWZHHQ &RQILGHQFH ,QWHUYDO IRUL M 0 HDQV /60 HDQ L /60 HDQ M Statistics, Data Analysis, and Modeling(1990, pp. 234-236), with response variablesCreatinine = amount of the pigmentcreatinine and Chloride = amount ofchloride in urine samples. These are to becompared for subjects in four differentobesity groups ("lighter underweight, "heavier underweight, "lighter obese, "heavier obese ), adjusting for a singlecovariate, Volume. There are six pairwisecomparisons of the amount of creatinine andsix of the amount of chloride. The family ofinferences will include all 12 confidenceintervals for the differences in means, allcovariate-adjusted. The data, code, andoutput are given in Display 2:Display 2.)