Zeliade White Paper May, 2011 Zeliade Systems

1 Zeliade SystemsModel validation : theory, practice and perspectivesZeliade White PaperMay, 2011 Zeliade SystemsZWP-0006 Zeliade Systems SASZ eliade Systems56 rue Jean-Jacques Rousseau75001 ParisFrancePhone : +(33) 1 40 26 17 29 Fax : +(33) 1 40 26 17 81e-mail validation : theory, practice and perspectivesaNUMBER:ZWP-0006 NUMBER OF PAGES:14 FIRST VERSION:January, 2010 CURRENT VERSION:May, 2011 Martini and P. Henaff (D epartement LUSSI, T el ecom Bretagne)This document must not be published, transmitted or reproduced without permission from Zeliade Systems 2007 2011 Zeliade Systems SAS. All Rights : Zeliade Systems is a registered trademark of Zeliade Systems SAS. All other company and product names referenced in this document are used for identificationpurposes only and may be trade names or trademarks of their respective disclaimer: Zeliade Systems SAS AND/OR ITS SUPPLIERS MAKE NO REPRESENTATIONS ABOUT THE SUITABILITY OF THE INFORMATION CONTAINEDIN THIS DOCUMENT FOR ANY PURPOSE.

2 THIS DOCUMENT IS PROVIDED AS IS , WITHOUT WARRANTY OF ANY KIND. Zeliade Systems SAS AND/OR ITSSUPPLIERS HEREBY DISCLAIM ALL WARRANTIES AND CONDITIONS OF WITH REGARD TO THIS INFORMATION, INCLUDING ALL IMPLIED WARRANTIES ANDCONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON INFRINGEMENT. IN NO EVENT SHALL Zeliade Systems SASBE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE IN THIS Nature of Model Risk1 Model validation teams2 Qualitative Assessment of Model Risk4 Derman classification ..4 When everything else fails ..5 Empirical Assessment of Model Risk6 Dynamic hedging simulation ..6 The calibration conundrum ..7 The need for simulation ..7 Quantification of Model Risk8 The robustness of Black Scholes.

3 8 The seminal work of Avellaneda and Lyons on model uncertainty ..9 Recent proposals and open issues ..9 Conclusion11In July of 2009, the Basel Committee on Banking Supervision issued a directive [21]requiring that financial institutions quantify model risk. The Committee further statedthat two types of risks should be taken into account: The model risk associated withusing a possibly incorrect valuation, and the risk associated with using unobservablecalibration parameters . The resulting adjustments must impact Tier I regulatory cap-ital, and the directive must be implemented by the end of the surface, this seems to be a simple adjustment to the market risk framework,adding model risk to other sources of risk that have already been identified withinBasel II. In fact, quantifying model risk is much more complex because the source ofrisk (using an inadequate model) is much harder to characterize.

4 Twelve months awayfrom the deadline, there is no consensus on this topic. There is fortunately a growingbody of literature, both from the academia and the industry, and the purpose of thispaper is to summarize the development of the notion of model risk and present thecurrent state of the art, before outlining open issues that must be resolved in order todefine a consistent framework for measuring model Nature of Model RiskFinancial assets can be divided into two categories. In the first category, we find theassets for which a price can be directly observed in the financial market place. These arethe liquid assets for which there are either organized markets ( Futures exchanges)or a liquid OTC market ( interest rate swaps). For the vast majority of assets,however, price cannot be directly observed, but needs to be inferred from observableprices of related instruments.

5 This is typically the case for financial derivatives whoseprice is related to various features of theprimaryassets, depending on a model. Thisprocess is known as marking-to-model , and involves both a mathematical algorithmand subjective components, thus exposing the process to estimation error. Beyond thisgeneric observation, the notion of Model Risk has been interpreted in at least threemanners. A first interpretation of Model Risk stems from the observation that various mod-els can be calibrated to perfectly price a set of liquid instruments, but produce in-consistent estimates for an exotic product. If one accepts that there is one true model, then model risk refers to the risk of mis-specification. A second interpretation focuses on the operational use of a model, that is used notonly to compute a price, but equally importantly to compute risk indicators fordynamic hedging.

6 In a perfect world where the true process for each risk factoris known, and where hedge instruments are readily available, we know that toc Zeliade Systems SAS1/14 ZWP-0006 May, 2011each derivative corresponds a dynamic replicating portfolio. Thus model-risk can be assessed by observing the hedging error, the discrepancy between thepayoff of a derivative and the value of the replicating portfolio. Well publicized events in the banking industry have highlighted a third manifes-tation of Model Risk. When liquidity is low, how should a product be marked tomodel so as to minimize the risk of discrepancy when a transaction can finallybe observed? To quote Rebonato [23]:Model risk is the risk of occurrence of a significant difference betweenthe mark-to-model value of a complex and/or illiquid instrument, andthe price at which the same instrument is revealed to have traded in summary, Model Risk can lead to both mis-pricing and mis-management of thehedging strategy.

7 Mis-pricing clearly will have the most spectacular consequences,but mis-hedging is an equally serious importance of Model Risk has gradually emerged in the industry: in first-tierbanks,Model Validationteams have been created as an independent power besidesfront-office trading/quant teams. These organizations are now briefly validation teamsThese teams are usually organized by asset class (equity, commodities, FX, fixed in-come in a broad sense, credit, cross-currency). The day-to-day duty of all these modelvalidation teams is to qualitatively and quantitatively validate the models used by thefront-office. A closer look, however, reveals that their roles vary widely across institu-tions: In its narrowest interpretation, model validation is an almost pure theoreticaland qualitative matter, which requires a solid mathematical finance backgroundand a good sense for the business domain.

8 It focuses on whether a model ormethodology is adequate for the purpose it is used for by the front-office: do not use local volatility models for pricing Cliquet options since they willbe sensitive to the forward volatility which the local volatility models do notencompass, do not use the SABR time-slice approximation for small strikes because itwill yield artificial arbitrage Zeliade Systems SAS2/14 ZWP-0006 May, 2011 In a broader definition, the role involves the validation not only of the models ormethodologies at hand, but also of theirimplementations. This involves review-ing the code provided by the front-office, running simulation stress tests, anddeveloping alternative implementations as a cross-check. This new dimensionbrings a fresh set of challenges however, as the validation team must now dealwith the varied technologies used by the front-office groups (Excel/VBA, Mat-lab, C++).

9 This challenge is compounded by the fact that these teams may havea very strong Quant culture, but usually less than a strong IT discipline. If theydecide to develop an in-house analytics library, they will often rely on third-partycomponents. A still wider interpretation of the role is to validate not only the model imple-mentation, but the wholeprocesswhere the model comes into play: What the model is used for: either getting a delta from a calibrated pricefor short-term arbitrage, or providing a reference price for an exotic productwhich is to stay in the book; whether the complex product at hand is hedgedin various ways with indicators provided by the model. In fact the rightpoint of view is very often to validate not a model, but a strategy performedby a market actor which involves model-computed quantities at some typical example in the Vanna-Volga strategy, mostly used on FX markets,which is not amodelin the traditional sense.

10 The market data and other data sources on which the strategy depends: doyou calibrate your models on the right data? For instance there are severaldifferent fields in Bloomberg for an option price: do you use the right one?Regarding dividends and repos on equity markets, where does your datacome from? How do you handle implied dividends and/or repos stemmingfrom the call/put parity in force on the vast majority of equity indices? At which horizon do you use the model which have been calibrated on 3-years market data? How is the model extrapolated? At which frequency do you re-calibrate your model? How do you handlethe discrepancy between the implied volatility/correlation markets and thehistorical ones? What happens in case of a Credit event for one of the counterparties?The academic literature has also recognized the issue early on, starting with contri-butions by Merton and Scholes themselves [18], shortly after the publication of theBlack-Scholes model.