Preparing for Basel II Modeling Requirements - Forecasting

1 As mentioned in the firstarticle in this series, thecurrent school of thoughtsurrounding the probability ofdefault (PD) and loss givendefault (LGD) models mentionedin Basel Consultative Papers isthat banks should have separatemodels for the obligor and thefacility. The obligor model shouldpredict the PD usually 90-plusdays delinquent or in foreclosure,bankruptcy, charge-off, reposses-sion, or restructuring. Models onthe facility side should predict theloss given default (LGD) or 1minus the recovery rate. In the initial article, logisticregressionwas the approach recom-mended for building PD statistical technique uses aset of explanatory variables whosevalues today would hopefully pre-dict a loan s probability of defaultsometime over the next 12months. On the LGD side, theapproach recommended was touse either linear regressionor tobitregressionto estimate the model . Directives from Basel IIParamount to using theadvanced approach as specified inthe Basel II Capital Accord is afocus on model validation.

2 TheNew Basel Capital Accord, pub-lished January 2001, includes thefollowing:302. Banks must have arobust system in place to validatethe accuracy and consistency ofrating systems, processes, and theestimation of PDs. A bank mustdemonstrate to its supervisor thatthe internal validation processenables it to assess the perform-ance of internal rating and riskquantification systems consistent-ly and The process cycle ofmodel validation must also include: ongoing periodic monitoringof model performance, includingevaluation and rigorous statisticaltesting of the dynamic stability ofthe model and its key coefficients; identifying and document-ing individual fixed relationshipsin the model that are no longerappropriate; periodic testing of modeloutputs against outcomes on anannual basis, at a minimum; and a rigorous change controlprocess, which stipulates the pro-cedures that must be followedprior to making changes in themodel in response to of yet, The New BaselCapital Accorddoes not givespecifics or standards related tothe validation process.

3 Introduction to ValidationsValidation includes issues ofdata quality, documentation, sen-sitivity analysis, model specifica-tion, sample design, the perform-ance of statistical tests, and thedevelopment of measures for50 The RMA Journal June 2003 Preparing for Basel IIModeling RequirementsPart 2: model Validationby Jeffrey S. MorrisonCAPITALREQUIREMENTSThis article, the second in a four-part series, discusses someapproaches to the validation of statistical models asrequired by the new Capital Accord. 2003 by RMA. Jeff Morrison is vice president, Credit Metrics PRISM Team, at SunTrust Banks Inc., Atlanta, accuracy. Although notminimizing the importance ofthese other areas, for brevity ssake the remainder of this articlewill focus on quantifying accuracymeasures. In this light, model val-idation simply refers to checkingthe accuracy of your model oversome specific period of time.

4 Howmany loans actually went intodefault during the year and whatdid their predicted default proba-bilities look like? If most of yourdefaults had a predicted probabili-ty of default near 10%, then yourmodel may be doing a poor only is the validationprocess part of Basel Requirements ,it is central to any model develop-ment process, regardless of itsapplication. Even econometricforecasting models models devel-oped using aggregated data witheconomic time series are validat-ed for accuracy. Credit-scoringmodels are also validated for accu-racy. Because Modeling is indeedan art, statistical algorithms aredeveloped and redeveloped until aformulation is found that reflectsthe most accurate results andmakes the most business sense. Validations can be done in avariety of ways, ranging from thesimple to the the validationonly on your model develop-ment the validation ona sample of accounts thatwere not used to develop themodel, but were taken fromthe same period of the validation ona single holdout sample fromtime periods outside yourmodel development a step-throughsim-ulation process across multi-ple time periods while recali-brating the first approach is the moststraightforward and is typicallyperformed as the model is devel-oped.

5 Here, the same data thatwas used for estimating the modelis used for validation. Althoughthis type of validation tends tooverstate the model s predictiveability, it may be necessary ifthere are a limited number ofdefaults available for model build-ing purposes. If there are sufficient defaultsavailable, the second method ispreferred. A random sample ofdata is held out from the modelestimation; the second methodruns the holdout data against themodel to compute its predictedvalues for validation method is widely used forvalidating a variety of differentmodels and serves as an aid to thestatistician in selecting the remaining methods aremore advanced not becausetheir techniques are necessarilymore complicated but becausethey require a greater depth ofdefault history. The thirdapproach holds out data for valida-tion from prior periods to see ifthe level of accuracy remains thesame from year to year.

6 This is anindication of how stable yourmodel may be over time. Thefourth approach is a combinationof validations and model recalibra-tions. The idea is to simulatemodel development and its pre-dictiveness over time given thatmodel revisions are done annuallyas new defaults are accumulatedand added to the the Obligor model :Probability of DefaultLet s assume you use the sec-ond validation method to evaluatethe accuracy of a PD model andhold out a sample of accounts thatwere not used in model develop-ment. So how many defaults doyou need? Generally speaking,hundreds of defaults are necessaryto properly test the model themore, the better. Since moredefaults are available in credit cardportfolios, thousands of defaultsare typically used in validations. To perform a validation, youneed predicted default probabili-ties from your model and a defaultstatus indicator showing whetherthe account defaulted or not.

7 Withthis information handy, you caneasily calculate two measures ofmodel accuracy where all youneed to be able to do is sort andadd. Following is a step-by-stepguide for PD 1: Create your holdoutsample, if 2: Code your you built yourmodel with a default indicator of 1 if the loan defaulted and 0 otherwise, make sure your defaultindicator in your holdout sample iscoded the same way. Step 3: Sort your the data from high-est to lowest, based on the proba-bility of default. If your data set issmall enough, you could even dothis in 4: Record minimumand maximum probabilities ineach 5% for Basel IIModeling RequirementsPart 2: model ValidationStep 5: Add some numberstogether. Now start totaling thedata into buckets at 5% intervalsfrom the top down. Produce thefollowing columns by bucket: Number of defaults. Number of 6: Calculate cumula-tive percentages by bucket: Cumulative number ofdefaults.

8 Cumulative number of 1 shows this processfor a fictional model in which thenumber of defaults was totaledinto 20 buckets, each represent-ing about 5% of the this procedure is appliedto an accurate model , the majori-ty of thedefaultersshould be accu-mulated in theearlier , thenondefaultersshould be foundtoward the bot-tom shows usthe power themodel has indistinguishingbetweendefaulters andnondefaulters. Step 7:Identify per-centages ofdefaulters andnondefaultersin the top twobuckets. Notethat the columnF labeled Cumulative % Defaults indi-cates that of the totaldefaulters in the holdout samplewere identified in the top 10%(two buckets) of the sorted bigger these numbers, thebetter. Note only of thenondefaulters were found, asshown in column G labeled Cumulative % Nondefaults. These measures, reflecting theaccuracy of the model for the top10% of the data, can serve as anexcellent way to validate compet-ing 8: Calculate the K-Svalue.

9 Another measure of accura-cy can be computed from columnsF and G. This value, called K-S,is simply the maximum differencebetween these two columns ofnumbers. The K-S value canrange between 0 and 100, with100 implying the model does aperfect job in predicting defaultsor separating the two general, the higher the K-S,the better the model . The placewhere that maximum occurs isthat point of maximum shown in Column H, the K-Svalue in this example is andoccurs at the 9th 9: Produce a is done by simply graphingcolumns F and G. A graphicaldepiction of this table, as seen inFigure 2, goes by a variety ofnames, such as power curve or RMA Journal June 2003 Figure 1 ABCD EFGH5%MinMax## of NonCumulativeCumulativeDifference inBucket ProbabilityProbabilityDefaultsDefaults% Defaults% Nondefaults% for Basel IIModeling RequirementsPart 2: model ValidationThe vertical axis is the cumu-lative percentage of defaulters ornondefaulters counted or identi-fied.

10 The horizontal axis reflectshow far down the sorted list youare. In other words, a value of 30on the horizontal axis means thatyou have examined the upper 30%of the validation data. The greyline at the top reflects the resultsof a theoretically perfect modelthat correctly predicts all thedefaults. That s the best you cando. The dark blue line shows thecumulative percentage of default-ers based upon your estimatedmodel (column F) while thelighter blue line shows the cumula-tive percentage of nondefaulters(column G). The black line in the middlereflects a na ve model that identi-fies defaulters simply at other words, this random model has no predictive infor-mation content. It is sometimesused as a benchmark when com-paring competing models. For astatistical model to have any valueat all, it must perform better thana random guess at who woulddefault. The K-S value of isthe vertical distance between thedark blue and lighter blue lines.