Transforming Credit Risk Management - Validating Calibration

Validation is a process to validate risk indicators’ calibration with risk drivers. Risk drivers must be calibrated with credit losses and actual measures need to be validated against estimated measures. The ultimate purpose is to validate credit losses measures. The validation process attempts to prove consistency and accuracy of measurements. Consistent validation can be across a time period or across the calibration to credit losses. Furthermore, the Basel Accord recommends that testing and validation methods should not systematically vary over the economic cycle.

Validation Consistency

• Validation with internal data encompasses using historical data over a long period covering a range of economic conditions and one or more economic cycles.
• Validation with external data encompasses
• For the observation period.
• Use of quantitative validation tools to validate the calibration by benchmarking the measurement against external data.
• Calibration needs to be corrected if realized values are more than expected. Cyclic variation may or may not be considered (depending upon view on the cycle).
• One of the validation methods can be benchmarking calibration against the supervisory estimates of risk drivers.

Validating Accuracy

The following table indicates validating accuracy and reliability of PD (probabiliuty of default):

Validation approaches differ for portfolios and usually follow the ‘bottom-up’ approach.

Take Note – Transformation Process

In a study released in April 1999, the Committee concluded that it was premature to consider the use of credit risk models for regulatory capital, primarily because of difficulties in calibrating and validating these models. Back-testing bank risk systems has been a key component of regulatory approaches to internal models since banks were first permitted to use internal models for calculating regulatory capital for market risks in the trading book in the 1996 amendment to the Capital Accord (implemented from 1998). Some of the validation methodologies are:

1. Performance Testing

This is similar to the back-testing concept for market risk (VaR) validation. It typically amounts to calculating observed default rates/loss rates in each rating category over a sufficient period of time and verifying the existence of a strong relationship between internal ratings and default/loss rates. Actual default/loss rates can then be compared to those assumed or predicted by the rating model. The main difficulty posed by this method is the need for a large series of internal ratings and default or loss events. Banks, therefore, adopt other methodologies too.

2. Scenario Analysis

The purpose of scenario analysis is to identify the financial impact of low probability, but iincludes plausible events that may not be captured by a statistically based model. Therefore, the use of a credit risk model should be supplemented by stress-testing the assumptions used.

3. Benchmarking

This is an efficient way of approaching the validation of internal ratings for publicly traded companies. It involves contrasting the output of an internal rating system against estimations of default/migration probabilities or losses obtained using other rating sources. Benchmarking is required against the following:

• Rating agencies output.
• Pool of database – supervisors need to take initiatives to establish such a database in their countries.
• Other model output.

Take Note – Risk Architecture and Best Practices – Convergence of ex-post with ex-ante

The (ex-ante) probability of default should not be significantly different from the (ex-post) realized default frequency. Some of the thumb rules to validate credit rating systems are:

• Ex-post default rates within any given rating category should be larger than that of a higher (i.e. better) rating category.
• Ex-post default rates should increase with the time horizon. It is obvious that the default rates of companies based on a time horizon of five years have to be equal or greater than those based on a time horizon of one year.
• For companies with outstanding corporate bonds, credit spreads may be compared to internal credit ratings.

External Assessment by:

• Credit rating agencies
• Consultants
• Supervisors – market test

One of the tests likely to be insisted by supervisors may be 30 – 60 per cent change in the PD. This change is equivalent to one or two notches downgrading of borrowers. It was found that this increases mean capital requirement by about eigth per cent to 15 per cent (0.64 – 1.2 per cent additional capital) of the capital required (of eigth per cent).

Multivariate stress tests are now becoming a reality. Until recently, stress tests considered only a single risk factor change. This implicitly means null correlation between the risk factors. In reality, however, simultaneous changes in the risk factors are observed. Univariate stress tests should therefore be supplemented by multivariate stress tests in which more than one risk factor is changed at a time. (For example, a multivariate stress test was conducted by a regulator in Germany for its banking industry.)

Minimum standards for a model:

• Clarity on the theory, assumptions, mathematical or empirical basis of the parameters, variables and data sources.
• Rigorous validation process.
• Identification of the circumstances under which a model might not work.
• Rigorous analysis to calibrate proxies with risks.
• Regular validation.
• Demonstration of estimates as representative of long-run experience.
• PD estimates must be a long-run average of one-year default rates for borrowers in the grade with the exception of retail exposures.

Best Practices for Retail Portfolio

Retail exposure, being more granular and possessing homogeneity, will have a lower loss volatility and the loss curve will be nearer to a normal distribution. Therefore, the shape of the curve is characterized by two statistics – mean and standard deviation of loss rates. However, it should be noted that curves built on the basis of past data are generally not a good fit because of:

• selection and underwriting methods changing during the period; and
• structural changes during the period.

Therefore the following steps need to be taken to build a good fit curve:

• Realistic assumption of account homogeneity.
• Greater flexibility in the “fitting” process when choosing the “best” distribution-type associated with the historical loss data.
• Greater use of cross-sectional data, if there are sufficient numbers of individual accounts in a segment to avoid the problem with distribution non-stability.
• Consider data from sufficiently longer periods.

Implicit in the assumption is that the retail assets are more diversified and, therefore, less correlated with the overall economy than the corporate portfolio. However, geographic concentration remains. There are no explicit adjustments for maturity in the retail mortgage portfolio. Going forward, both these adjustments may be made by banks in their models. Credit risk based on one-year default rates does not reflect credit risk on mortgages.

Interest rate risks and the choice of options though is outside the discussion of this paper but has definite impact on mortgage lending type assets. Banks are expected to implement systems designed to validate ratings and bank supervisors will check the efficiency of those validation systems.