The validation of Credit Rating and Scoring Models

1 The validation processLiterature reviewMethodological proposalsThe validation of Credit Rating and ScoringModelsRaffaella of Milano-BicoccaSwiss Statistics MeetingGeneva, SwitzerlandOctober 29th, 2009 Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsOutline1 The validation process2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve3 Methodological proposalsCurve of Classification Error Costs and Error CostsRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsOutline1 The validation process2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve3 Methodological proposalsCurve of Classification Error Costs and Error CostsRaffaella

2 CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsOutline1 The validation process2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve3 Methodological proposalsCurve of Classification Error Costs and Error CostsRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsPreliminary definitionsCreditratingandscoringmodels estimate the creditobligor sworthinessand provide an assessment of theobligor s future powerof a Rating or Scoring modeldenotes its ability to discriminateex antebetweendefaulting and non-defaulting assesses the discriminatory powerof a Rating or Scoring modelRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsPreliminary definitionsCreditratingandscoringmodels estimate the creditobligor sworthinessand provide an assessment of theobligor s future powerof a Rating or Scoring modeldenotes its ability to discriminateex antebetweendefaulting and non-defaulting assesses the discriminatory powerof a Rating or Scoring modelRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological

3 ProposalsPreliminary definitionsCreditratingandscoringmodels estimate the creditobligor sworthinessand provide an assessment of theobligor s future powerof a Rating or Scoring modeldenotes its ability to discriminateex antebetweendefaulting and non-defaulting assesses the discriminatory powerof a Rating or Scoring modelRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveEach borrower is characterized by two random variables:the scoreSassigned to the borrower is a continuous r. support( , )the Bernoulli the borrower s state at theend of a fixed time-periodB={1,the borrower s state is default (d);0,the borrower s state is non default (n).}

4 The conditional distribution functions ofSgiven a value ofBare denoted respectively byFd( )andFn( ).Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveEach borrower is characterized by two random variables:the scoreSassigned to the borrower is a continuous r. support( , )the Bernoulli the borrower s state at theend of a fixed time-periodB={1,the borrower s state is default (d);0,the borrower s state is non default (n).The conditional distribution functions ofSgiven a value ofBare denoted respectively byFd( )andFn( ).}

5 Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveEach borrower is characterized by two random variables:the scoreSassigned to the borrower is a continuous r. support( , )the Bernoulli the borrower s state at theend of a fixed time-periodB={1,the borrower s state is default (d);0,the borrower s state is non default (n).The conditional distribution functions ofSgiven a value ofBare denoted respectively byFd( )andFn( ).Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveThe distribution function of the scoreSisF(s) =pFd(s) + (1 p)Fn(s)wherepis the probability of defaultp=P[B=d].}

6 Theaccuracy(AC) isAC=pFd(s) + (1 p)[1 Fn(s)] =2pFd(s) F(s) + (1 p)Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveThe distribution function of the scoreSisF(s) =pFd(s) + (1 p)Fn(s)wherepis the probability of defaultp=P[B=d].Theaccuracy(AC) isAC=pFd(s) + (1 p)[1 Fn(s)] =2pFd(s) F(s) + (1 p)Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveActual defaultActual non defaultPredicted defaultTrue Default (TD)False Default (FD)(below s)Type II errorPredicted non defaultFalse Non Default (FN)True Non Default(above s)Type I error(TN)NdNnhit rate Fd(s) =TDNdfalse alarm rate Fn(s) =FDNn F(s) = p Fd(s) + (1 p) Fn(s)

7 =TD+FDNd+Nnwhere p=NdNd+NnRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveActual defaultActual non defaultPredicted defaultTrue Default (TD)False Default (FD)(below s)Type II errorPredicted non defaultFalse Non Default (FN)True Non Default(above s)Type I error(TN)NdNnhit rate Fd(s) =TDNdfalse alarm rate Fn(s) =FDNn F(s) = p Fd(s) + (1 p) Fn(s) =TD+FDNd+Nnwhere p=NdNd+NnRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveActual defaultActual non defaultPredicted defaultTrue Default (TD)False Default (FD)(below s)Type II errorPredicted non defaultFalse Non Default (FN)True Non Default(above s)Type I error(TN)NdNnhit rate Fd(s) =TDNdfalse alarm rate Fn(s) =FDNn F(s) = p Fd(s) + (1 p) Fn(s)

8 =TD+FDNd+Nnwhere p=NdNd+NnRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveActual defaultActual non defaultPredicted defaultTrue Default (TD)False Default (FD)(below s)Type II errorPredicted non defaultFalse Non Default (FN)True Non Default(above s)Type I error(TN)NdNnhit rate Fd(s) =TDNdfalse alarm rate Fn(s) =FDNn F(s) = p Fd(s) + (1 p) Fn(s) =TD+FDNd+Nnwhere p=NdNd+NnRaffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)curve:CAP(u) =Fd[F 1(u)],u (0,1)Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)curve.

9 CAP(u) =Fd[F 1(u)],u (0,1)Raffaella CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)synthetic index(BCBS, 2005):AR=aRaR+aQAR [0,1]optimal cut-off score(Hong, 2009): the intersection of theCAP curve and the iso-performance tangent lineFd(s) =12p[F(s) +AC+p 1]drawbacks:-dependence on the sample relative frequency of defaultedborrowers;-the type II error and the costs of wrong classification CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)synthetic index(BCBS, 2005):AR=aRaR+aQAR [0,1]optimal cut-off score(Hong, 2009): the intersection of theCAP curve and the iso-performance tangent lineFd(s) =12p[F(s) +AC+p 1]drawbacks:-dependence on the sample relative frequency of defaultedborrowers.

10 -the type II error and the costs of wrong classification CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)synthetic index(BCBS, 2005):AR=aRaR+aQAR [0,1]optimal cut-off score(Hong, 2009): the intersection of theCAP curve and the iso-performance tangent lineFd(s) =12p[F(s) +AC+p 1]drawbacks:-dependence on the sample relative frequency of defaultedborrowers;-the type II error and the costs of wrong classification CalabreseValidation of internal Rating systemsThe validation processLiterature reviewMethodological proposalsCumulative Accuracy Profile CurveReceiver Operating Characteristic CurveCumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)synthetic index(BCBS, 2005):AR=aRaR+aQAR [0,1]optimal cut-off score(Hong, 2009): the intersection of theCAP curve and the iso-performance tangent lineFd(s) =12p[F(s) +AC+p 1]drawbacks:-dependence on the sample relative frequency of defaultedborrowers.