Business models in banks have undergone a material change thanks to the ‘good times’ the economy saw in the earlier part of this decade. Traditionally conservative banks stepped on the accelerator to tap the increasing financial requirements from corporate and retail customers. Credit analysts, who used to analyse each funding requirement on a case-to-case basis, now use models to vindicate (and in some cases to substitute) credit analysis. With continuous innovation in financial instruments, the market risk managers are continuously searching for sophisticated models to measure market risk accurately.
What seems to be astounding is that risk models have now become an integral part of decision making. Despite the viewpoint held by critics that risk models are just mathematical gimmicks, they have found a strong foothold due to their attributes of simplicity, comprehensiveness, objectivity, and their ‘perceived’ ability to accurately quantify risk. For instance; in corporate and consumer credit business, models are now being used for selecting target market, analysing creditworthiness, monitoring loan portfolios, and improving collections. In the investment banking and trading business, value at risk (VaR) has leapt from being a mere internal reporting measure to now being formally recognised by regional regulators.
Many variants of VaR measure, such as cash flow at risk (CFaR), credit value at risk (CVaR), and liquidity at risk (LaR) are also being developed. The prime motivation for using models is achieving consistency in risk measurement, complementing (if not driving) risk analysis, and having an acceptable measure of risk across the enterprise. Output from these models impact capital adequacy calculations and therefore the optimum capital to be set aside for unforeseen events. An underestimation would compromise the bank’s true capital adequacy and an overestimation would jeopardise the return on capital. With an increasing number of banks expanding overseas, it has become essential to align these risk models to the global best practices. This view has also been actively promoted by the regulators.
The credit crisis raised several questions on the effectiveness of models in measuring risk and predicting default. It is also believed that these models were responsible for creating billions of dollars of toxic assets resulting from credit derivatives based on sub-prime mortgages. Experts opine that the scale of financial crisis has been higher because too much reliance was placed on risk quantification models. While the credit risk models failed to correctly predict the credit losses, the market risk models failed to estimate worst possible losses in the case of large deviations from the normal. In some cases, market risk models were unable to measure risk for complex derivatives products. The capital set aside by banks for unforeseen losses was insufficient, causing increasing defaults by banks on their counterparty arrangements that caused a domino effect on the entire banking industry.
Most of these failed models were flawed at the outset because they assumed normal distributions, efficient markets, and ignored the inter-linkages between different markets and risks. The urgency to avoid committing such mistakes in the future underscores the need of taking up a systematic model substantiation process. Do these models provide an accurate picture of the risk position with respect to a credit or a market portfolio? Do the outputs of these models justify the credibility these models hold? Can banks rely on models for decision-making? Why did these risk models grossly understate risks and is it possible that such instances can be repeated in the future?
The answer to all these questions depends on the sturdiness of the qualitative and quantitative processes surrounding the risk model. The accuracy of risk models is based on the validity of their underlying assumptions and is therefore limited by these. The output of the models would remain consistent and acceptable only as long as the data churned remains consistent with the data used at the time of inception of the risk model. Relevance of the models is impacted significantly if they are not calibrated to fit the idiosyncratic factors and the risk environment in which the bank operates. The models currently used in banks are based on the simplified assumptions of normalcy and the scenarios generated are based on ‘normal’ distribution assumptions.
Due to this, the probability of causing extreme losses is always underestimated. As such, these models cannot adequately capture extreme market moves. Indigenously created credit risk models which are tested on the bank’s historical default data alone are more likely to throw an incorrect result if an ‘out-of-sample’ data is processed. If objectives of risk models are to be achieved, it is important to regularly correct these models before they could report risk incorrectly. Considering the damage that can be caused to a bank’s bottom line and its reputation, it is unwise to now ignore this subject.
Constituents of an erroneous model
Before applying a corrective action to these models, it is important to first understand what constitutes an erroneous model output and then investigate the causes of a model failure. The most common and effective method to appraise performance of a model is by back testing, i.e. comparing the output of the model with actual outcomes. There are three levels of errors that a model is likely to commit. An inaccurate model can be judged by the number of ‘bad’ applications it recommended and by its inability to determine which customers or instruments are more risky as compared to the rest. If a model is unable to provide such a judgment, further reliance on such models is not recommended.
At a second level, the model is evaluated on the basis of the number of instances of the model output breaching the performance thresholds set during model inception phase. In case of too many breaches, it can be concluded that the model needs further calibration in order to improve its accuracy. Reliance on such models should be limited.
At a third level, a model is judged through its ability to factor macro-economic variables into its analysis and its ability to exploit connections with other models, such as the loss forecasting models in case of credit risk. If a model fails this test, it is recommended that a risk analyst synthesise outputs from different models before making a decision. Through this review, a bank should also evaluate the severity of breaches and the precise reasons for its failure.
Value Proposition of Model Validation
Model validation (Figure 1) – the practice of assessing the assumptions, underlying theoretical basis, and data used by models, as well as the processing, output, and reporting – is one of the ways to manage, mitigate, and control model risk. Validation includes review of model usage relative to management’s defined objectives and the comparison of model methodologies to industry best practices. Through model validation, banks can assess the ongoing appropriateness of their models in the context of the latest market developments, industry trends, risk strategy, and preferences. In order to avoid potential conflicts of interest, it is important to have an independent assessment of the models and their use, a role that has been overlooked for a long time in this region.
The Basel II regulation has several references to model validation. Para 718 (XCix) lists out detail model validation requirements. Compliance to these principles alone will allow banks to report capital adequacy based on their internally created risk models. These requirements surround the need of having a regular cycle of model performance review, data base standards, testing of model outputs against outcomes, and the type of tests that the banks are required to conduct. Paragraphs 500-505 also state expectations of a model validation programme at banks for credit risk models. However the principles can be applied to market and operational risk models.
Figure 1: Model Validation Process
A detailed model validation exercise includes review of the following components:
Model objectives
Risk measurement models can be developed internally or can be purchased from a third-party vendor. In most cases banks are motivated to use risk models more by competition than by an identified need to have risk models. In such situations, the objectives to be accomplished are not well identified and are vaguely defined. This impacts acceptability, use, and relevance of models in daily operations. In the absence of well-determined model objectives, the indicators for measuring model performance are elementarily defined or not defined at all. Model validation starts with understanding the business needs of the bank, checking if the model objectives are aligned to the business needs, and developing key performance indicators appropriately. It is important to clearly define what is not being modeled as this ensures that the scope of the model is well understood in the bank.
Data churned
A risk model’s performance is directly related to the quality of the data processed. In case a bank does not have a structural solution to solve data quality issues, this issue becomes even more pertinent – it could result in a lot of repetitive, erroneous, and irrelevant data getting processed. In cases where data is updated dynamically from other models or applications, any calculation error on part of such systems or models will compromise quality of data. The model validation process includes assessment of data integrity, creation of information flow diagrams to understand key conversions and interfaces, review of risk database, and review of documentation procedures surrounding data to be gathered as per the model requirements. In case risk models are purchased from a third party, this process additionally involves review of data mapping of the bank’s internal database structure to the data model as required by the vendor application.
Model methodology
Methodology covers principles governing the risk profile of the data, assumptions driving model functioning, and data calibration. The data that would eventually go into the model as input would have its own risk profile. If data used for model creation comes from a period of high growth, the model will not provide appropriate results in a low or stable growth situation. Similarly, if the data comes from a period of low growth, the model would fail to detect the risk of decisions in normal or high growth periods. Analysis of the risk profile of the data is extremely important as it directly impacts the consistency of model output through the business cycle. One way to achieve this is to discount the data using growth rates in order to remove any bias.
Models function on the basis of certain theoretical and practical assumptions. These assumptions are critical input elements that have been reviewed and approved by management as being appropriate for use by the model. These assumptions must be regularly compared to observable reference points or other external sources such as the market data, historical data, industry research, and academic research to check if the assumptions are indeed valid and up-to-date.
Additionally, model validation also covers the reviewing of model parameters, analysing the impact of each parameter on the result, studying the volatility of data required by each parameter, and measuring the correlations across different parameters. For instance, for a credit risk model, if it is observed empirically that the current ratio and the working capital ratio are highly correlated, through model validation; an effort will be made to eliminate this redundancy. Through such analysis, data range for each parameter is derived. In case of models that are acquired from a third party, calibration of models as per the bank’s level of risk must be performed. Without such a calibration, the model can provide incompatible results.
Processing and outcome analysis
Most risk models are either converted into applications or into spreadsheets. Data processed through such applications or spreadsheets must provide expected results. Through model validation, a detailed review of the model’s processing component over multiple scenarios or conditions is conducted through stress testing and scenario analysis. In order to check the robustness of the model, the model is back tested using the bank’s historical data and the results are then compared with the current portfolio position. Depending on the results of such tests, data range for each parameter is determined and compared with the actual parameter range. Through this process, the risk model’s constraints are also identified and these are formally documented. The underlying logic and calculations of the model to assess the model’s internal integrity for performing the calculation for future periods is also affirmed in this process. In case the model is converted to an application, a robust code review is also performed.
Model governance and performance measurement
Some of the banks in the United Arab Emirates (UAE) use risk models to either provide a risk rating for compliance purpose or to map price against the risk of the proposition. Subjective risk analysis is a more influencing factor in making decisions and contribution of models to decision-making here is only partial. In such a situation, relevance of risk models and consequently model maintenance is not given a high priority. It is for this reason that the organisation structure and decision making process surrounding the risk model need to be looked into.
A model’s performance should be regularly compared to the set of performance standards. Regular back testing of the model’s output should be conducted to check any abnormal deviations. Governing policies must provide for active reporting of the model’s performance to the relevant stakeholders in the bank. Through model validation, standards set for model development, change management, and validation are reviewed. A bank can test transparency of its data flows and process, comprehensiveness of documentation, board and senior management roles and responsibilities, quality of resources, and key personnel risks through a detailed model validation exercise. Through such reviews, banks can obtain deeper insights into causes of any unexpected behavior of the model. Such performance reviews often are the source of further changes to model objectives and methodology.
Conclusion
What we have learnt from the humbling past couple of years for the banking industry is that risk models are valuable tools and modern finance would be limited without them – however, without their appropriate use, they tend to become out-of-date and susceptible to provide inappropriate results with sometimes major consequences. It would be incorrect to pass on the blame entirely on risk models for not being able to warn beforehand, for the devil does not lie in the model, but in the fallible model infrastructure. The current recessionary condition has given risk managers refueling time which should be used to re-establish credibility in risk models by creating a robust and yet adaptable model validation structure. A risk model is not a ‘black box’ and it is necessary that the banks realise that with the changing role and scope of risk management, the underlying risk models must also evolve. We have passed the stage of creating sophisticated models, but we are yet to arrive at the stage of effective use and validation of risk models.