Basel Committee on Banking Supervision Working Paper No. 14

1 Basel Committee on Banking Supervision Working Paper No. 14 Studies on the Validation of Internal Rating Systems Revised version May 2005 Studies on the Validation of Internal Rating Systems The Working Papers of the Basel Committee on Banking Supervision contain analysis carried out by experts of the Basel Committee or its Working groups. They may also reflect work carried out by one or more member institutions or by its Secretariat. The subjects of the Working Papers are of topical interest to supervisors and are technical in character. The views expressed in the Working Papers are those of their authors and do not represent the official views of the Basel Committee , its member institutions or the BIS. Copies of publications are available from: Bank for International Settlements Information, Press & Library Services CH-4002 Basel , Switzerland Fax: +41 61 / 280 91 00 and +41 61 / 280 81 00 This publication is available on the BIS website ( ).

2 Bank for International Settlements 2005. All rights reserved. Brief excerpts may be reproduced or translated provided the source is stated. ISSN 1561-8854 Studies on the Validation of Internal Rating Systems Chairman of the Validation Group Mr Thilo Liebig Deutsche Bundesbank, Frankfurt Members of the Validation Group Mr Vincent Baritsch Financial Services authority , London Ms Rosalind L Bennett Federal Deposit Insurance Corporation, Washington Mr Martin Birn Bank for International Settlements, Basel Mr Stefan Blochwitz Deutsche Bundesbank, Frankfurt Mr Jaap W B Bos De Nederlandsche Bank, Amsterdam Mr Ben Carr Financial Services authority , London Ms Eva Catarineu Banco de Espa a, Madrid Mr Alvin Chow monetary authority of singapore , singapore Mr Klaus D llmann Deutsche Bundesbank, Frankfurt Ms Antonella Foglia Banca d Italia, Rome Mr Nils Chr Framstad Kredittilsynet, Oslo Mr Erik A Heitfield Board of Governors of the Federal Reserve System.

3 Washington Mr Jon S Hellevik Kredittilsynet, Oslo Mr Mark Levonian Office of the Comptroller of the Currency, Washington Mr Soon Chong Lim monetary authority of singapore , singapore Ms Nancy Masschelein National Bank of Belgium, Brussels Mr Hiroaki Miyagishi Bank of Japan, Tokyo Mr Gregorio Moral Banco de Espa a, Madrid Mr Vichett Oung Commission Bancaire, Paris Mr Yasushi Shiina Financial Services Agency, Tokyo Mr Dirk Tasche Deutsche Bundesbank, Frankfurt Mr Satoshi Yamashita Financial Services Agency, Tokyo Studies on the Validation of Internal Rating Systems i Table of Contents Executive summary ..1 Dynamics of rating systems ..2 Validation of PD, LGD and EAD ..3 Outlook ..4 Studies on the Validation of Internal Rating Systems ..7 I. Key components of validation ..8 Structure ..9 II. Dynamics of rating systems ..10 Characteristics of obligor-specific default Classification of rating systems.

4 14 Dynamics of pooled PDs ..16 Quantifying pooled Historical default experience ..18 Statistical models ..19 External mapping ..19 Appendix: An illustrative model of rating system dynamics and PD Introduction ..21 Obligor default model ..21 Risk Pooled PDs ..24 Backtesting pooled PDs with observed default frequencies ..25 Conclusions ..27 III. Rating and probability of default Discriminatory power ..30 Calibration ..32 Open issues ..35 Appendix: Statistical measures of discriminatory power and calibration Cumulative Accuracy Profiles and Accuracy Ratio ..36 Receiver Operating Characteristic and the area under the Receiver Operating Characteristic curve ..37 Connection between Accuracy Ratio and Area under the Curve ..39 Confidence Intervals for the AR and the area under the ROC ..39 ii Studies on the Validation of Internal Rating Systems The Pietra Index .. 42 Bayesian error rate.

5 43 Entropy measures .. 43 Kendall s and Somers 45 Brier score .. 46 Binomial test .. 47 Chi-square (or Hosmer-Lemeshow) test .. 52 Simulation study on the performance of the normal and traffic lights tests .. 53 IV. Loss given default 60 Introduction ..60 Definition and measurement of LGD .. 60 Definition of LGD .. 61 Definition of 63 Definition of loss .. 64 Some issues related to LGD estimation .. 65 Workout LGD .. 66 Components of workout LGD .. 66 Recoveries .. 67 Costs 67 Discount rates .. 68 Validation of 69 Conclusions .. 72 Areas for further 73 Appendix: Summary of empirical studies of Loss Given Default .. 74 Survey of empirical research .. 74 V. Exposure at default 94 Fixed or floating rate .. 94 Revolving or non-revolving .. 94 94 Restructuring .. 95 Obligor-specific characteristics .. 95 VI. 96 96 Objectives of benchmarking .. 96 Selection of benchmarks.

6 98 Mapping to the benchmark or to a master 99 Default models equivalence and 101 Studies on the Validation of Internal Rating Systems iii Rating dynamics and benchmarking ..101 Conclusions ..102 Areas for further research ..102 Appendix 1: A state space model for benchmarking ..103 Appendix 2: Application to implementation: a dynamic benchmarking A model of capital requirements dynamics ..105 An example of application: Inferring rating systems dynamic properties using dynamic benchmarking ..107 References ..111 Studies on the Validation of Internal Rating Systems 1 Executive summary In June 2004, the Basel Committee on Banking Supervision issued a revised framework on International Convergence of Capital Measurement and Capital Standards (hereafter Basel II or the revised Framework ). When following the internal ratings-based (IRB) approach to Basel II, Banking institutions will be allowed to use their own internal measures for key drivers of credit risk as primary inputs to their minimum regulatory capital calculation, subject to meeting certain conditions and to explicit supervisory approval.

7 In light of the need under Basel II for banks and their supervisors to assess the soundness and appropriateness of internal credit risk measurement and management systems, the development of methodologies for validating external and internal rating systems is clearly an important issue. More specifically, there is a need to develop means for validating the systems used to generate the parameters (such as PD, LGD, EAD and the underlying risk ratings) that serve as inputs to the IRB approach to credit risk. In this context, validation comprises a range of approaches and tools used to assess the soundness of these elements of IRB systems. In anticipation of the need for more knowledge regarding validation methodologies, in 2002 the Research Task Force (RTF) formed a subgroup (the Validation Group) to review and develop research on the validation of rating systems that would be useful to banks and supervisors as they consider options for implementing Basel II.

8 The work of the Validation Group collected in this volume of studies addresses a number of topics on rating system validation, with a particular focus on empirical validation methods. The Validation Group consists of representatives from eleven The main objectives of the project have been: to classify rating systems and their dynamic properties, and to develop a common terminology for validation purposes, to review validation methodologies that are currently applied in bank practice, and to analyse validation methodologies for the three key risk components probability of default (PD), loss given default (LGD) and exposure at default (EAD) from a theoretical perspective. Importantly, the collection of studies presented here is not intended to represent a comprehensive survey of all available validation methods and processes.

9 Rather, the RTF expects that this research will provide a valuable input to the work of the Basel Committee s Accord Implementation Group (AIG), national supervisors, and banks as they develop approaches for validating the risk parameters and rating systems needed to implement Basel II. In addition, it should be emphasised that these studies largely reflect the views of individual authors, and should not be viewed as representing specific Basel Committee guidance for national supervisors or financial institutions. Although validation is foremost the responsibility of banks, both bank risk managers and bank supervisors need to develop a thorough understanding of validation methods. Supervisors will need to review banks validation processes, and may also need to employ 1 Participating countries are Belgium, France, Germany, Italy, Japan, Netherlands, Norway, singapore , Spain, the , and 2 Studies on the Validation of Internal Rating Systems validation methods in evaluating whether banks rating systems comply with the operating standards set forth by Basel II.

10 Some validation methods, such as benchmarking risk parameters across banks, may be more practical for supervisors to implement than banks. The focus of the research in this collection has been on validation methods in general, without regard to whether those methods are implemented by banks or their supervisors. The key results of the studies in this volume are summarised below. When reading the summary and the original papers, it is important to understand that validation methodologies applied in the Banking industry are still preliminary in nature. However, encouraged by the focus of Basel II on rating systems, it is also fair to observe that validation has received the attention of both the industry and academics, as reflected in a growing stream of literature on this topic. Many open issues remain; some are conceptual, some are caused by insufficient data, and some may be better treated as technical aspects within the implementation process.