ROC CURVE ANALYSIS USING SAS - phusewiki.org

1 Zheng YaoSr. Statistical ProgrammerROC CURVE ANALYSIS USING SAS2 Outline Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary3 Outline Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary4 Background Biomarkers ( PD-1/L1) draw lots of attention nowadays. It is often of interest to use biomarker for disease screening, diagnosis and prediction. The fundamental for use of biomarkers in clinical practice is the accuracyand the optimalcut-off point selection The receiver operating characteristic (ROC) CURVE is a procedure that can aid in the accuracy assessment, ROC CURVE comparison and cut-off point selection.

2 5 BackgroundNote: TP=True Positive, FP=False Positive, FN=False Negative, and TN=True Negative. Sensitivity: the proportion of positive observations that are measured as positive, true positive rate (TPR), Sensitivity = a / (a + c) Specificity: the proportion of negative observations that are measured as negative, true negative (TNR), Specificity=d / (b + d); Youden s Index: (sensitivity + specificity) -1 Golden Standard OutcomeCut Point OutcomePositiveNegativePositiveTP (a)FP (b)NegativeFN (c)TN (d)Golden Standard OutcomeCut Point OutcomePositiveNegativePositive6020 Negative4080 Total100100 Sensitivity = 60 / (60+ 40) = Specificity = 80 / (20+ 80) = Youden s Index= (sensitivity + specificity) 1 = + -1 = ROC(Receiver-Operating Characteristic) CURVE : constructed with sensitivity (FP) on the vertical axis and 1-specificity (TP) on the horizontal Area under CURVE (AUC).

3 A measure of overall accuracy ROC curves comparison Cut-off point selection Youden s Index: (sensitivity + specificity) -17 Outline Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary8 Compare ROC curvesNOTE: Partial list output of the BOR dataset (obs=5) 156 cases of patients with positive PD-L1 expression were selected and treated with anti-PD-L1 agents (drug A). To e xp l o r e s t h e p o t e n t i al asso ci at i o n b e t w e e n PD-L1 expression in tumor tissue and anti-PD-L1 response ( BOR, PFS or OS) The expression of PD-L1 was measured with with two method (TCand IC) for each case Best Overall Response (BOR) is used as the predicitve measure according to (RECIST , 1).

4 The BOR variable is coded as 1 and 0 ( effective and noneffective) .BORPD-L1expressionEffective Non-effectivePositiveTP (a)FP (b)NegativeFN (c)TN (d)9 Compare ROC curves#:for example: Fertility: Hand in hand in Europe, GM: Ferrara IME program*Foruminclude scientific workshops and HCP trainings Is IC better than TC?10 Compare ROC curvesproc logistic data= BOR;modelBOR(event='1') = TC IC;roc"TC" TC; roc"IC" IC; roccontrastreference("TC")/estimate;run; The PROC LOGISTIC procedure for ROC CURVE comparison TC and ICare both independent variables in the model statement.

5 The ROCstatement produces a ROC the ROCCONTRAST statement produces a significance test for the ROC CURVE . the REFERENCE( TC ) statement means that TC is set as a reference when comparing with IC in the significance Reference code for ROC CURVE comparison11 Compare ROC curvesResults from Logistic model The significance test demonstrates that TC (p= ) and IC (p< ) are statistically significant for use in ROC CURVE . Logistic regression equation: nnxxppbbb ++= -1101ln12 Compare ROC curvesAUC statistical test and ROC CURVE ROC CURVE : all of three ROC curves are above the diagonal line.

6 AUC: all of three 95% CIs do not contain Therefore, we can conclude that all these three AUC are significantly better than ROC curvesROC comparison test IC vs. TC (p= ). Therefore, the AUC of IC ( ) is statistically larger than that of TC ( ). the PD-L1 scoring methodology of IC is better than 14 Outline Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary15 The PROC LOGISTIC procedure for ROC CURVE ANALYSIS The OUTROC=option creates a dataset containing sensitivity and specificity data which here is called ROCDATA.

7 The ROCstatement produces a ROC the ROCCONTRAST statement produces a significance test for the ROC CURVE . The PREDICTED=option creates a dataset containing estimated event probabilities ( pred) for each subject. proc logistic data= BOR;modelBOR(event='1') = IC/outroc=rocdata;output out=predpredicted=pred;roc "BOR"; roccontrast;run;Select a rational cut-off point in ROC CURVE analysis16 Select a rational cut-off point in ROC CURVE analysisThe ROC CURVE The diagonal line, from (0,0) to (1,1), is indicative of an independent variable that discriminates no different from guessing (50/50 chance).

8 The AUCis as compared to that of the diagonal line which is always 17 Select a rational cut-off point in ROC CURVE analysisAUC statistical test The 95% confidence interval ( , ) does not contain , therefore our AUC is significantly better than chance. A Chi-square test provides a p-value (p< .0001) associated with the null hypothesis (AUC = ). Note. If the 95% confidence interval does not include , then we can conclude that ROC CURVE is statistically significant. 18 NOTE: The model s intercept( ) and regression coefficients ( ) are needed for computing the cutoff model s intercept and regression coefficients Partial output from the PROC LOGISTIC procedure Logistic regression equation: nnxxppbbb ++= -1101lnSelect a rational cut-off point in ROC CURVE analysis19 Select a rational cut-off point in ROC CURVE analysisDetermine an optimal PD-L1 cutoff point for BOR Logistic regression equation: = intercept+ slope(X) = + *cutoffNote.

9 Partial list output of the rocdatadatasetnnxxppbbb ++= -1101ln -pp1ln20 Select a rational cut-off point in ROC CURVE analysisDetermine an optimal cutoff point for BOR Logistic regression equation: logit = intercept+ slope(X) = + *cutoffNote. The cutoff variable is formed by re-arranging logistic regression model to solve for X. The model is: logit = intercept + slope(x). nnxxppbbb ++= -1101lndatarocdata2(keep=cutoff probSensitivity Specificity Youden);setrocdata;logit=log(_prob_/(1-_ prob_));cutoff=(logit+ ) ;prob= _prob_;Sensitivity = _SENSIT_;Specificity =1-_1 MSPEC_;Youden= _SENSIT_+ (1-_1 MSPEC_)-1;run;Note.

10 Partial output of cutoff value and YoudenIndex by sorting the dataset in descendingYouden. A large Youdenmay be one criteria for deciding an appropriate cutoff Background Examples: Accuracy assessment Compare ROC curves Cut-off point selection Summary Accuracy assessment ROC CURVE comparison Cut-off point selectionBACK UP SLIDES