Sunday, December 18, 2011

Auditing Derivatives: Model Validation


This article appeared in The Malaysian Accountant journal, Sep-Oct 2011 issue
Auditing Derivatives: Think of what can go wrong - Model Validation
By Jasvin Josen
Auditing derivatives, in all likelihood, is one of the most challenging areas for an auditor. I focus on thinking of what can go wrong rather than just a standard review, for, derivatives are very flexible products and thus have a great potential to flourish in a reckless way when unguarded by proper supervision and regulation.
In the last five articles, we have observed the different issues that can surface as a derivative instrument moves along the trading floor and the Controlling Group, under the watchful eyes of Risk Management (with its Value-at-Risk model) and the Valuation team.
I must introduce a last function called Model Validation. This function has long been established in investment banks in the U.S. and Europe but possibly only now taking root in Asian investment banks.


This article will discuss the Model Validation role and the key points that the auditor should be mindful of, in his examination.
The Model Validation function
The main role of the Model Validation team is to ensure that the pricing models developed in-house or purchased from external vendors are credible. They want to be certain that the model is stable in every way, when giving out prices and risk measures. The team usually report to the Risk Management Group or it forms part of the Risk Management team.
So, what can go wrong?


The pressure
In order to be convinced that pricing models are stable and robust, the Model Validation team develops independent models for comparison. If they are satisfied with the results, the second phase is to perform vigorous tests such as back tests, stress tests and bug testing on the original model to determine its points of failure, if any.
There are several things that can go wrong. The more complicated the derivative, the more challenging this task becomes. E very parameter has to be tested to see if the model will collapse when the parameter goes out of range. But some parameters are “derived parameters”, i.e. these
parameters are calibrated from standard products in the market. The concern here is that not all market environments are considered when the calibration is performed.
This team has a major responsibility in representing to its bank that a model can be safely used. The team’s testing jobs can go on forever as they will always be something that can fall over in the model. For this reason, they will find an effective way to detain the main risks in the most efficient time. The concern here is that when the line is drawn, it is hoped that the procedure is able to catch all the significant and possible flaws.
Time pressure is always a concern. Banks usually need the model to be approved before a new trade can be priced or booked in the system. The Model Validation team will be under pressure to get back with the results fast so that the bank does not lose the potential client. The risk here is important insights can get lost when the team has a tight or sometimes unreasonable deadline.
The auditor will want to look into the skill and capabilities of the team. He will also want to look into the number of models tested within a time frame and compare it to previous times or other banks to get an idea if the team is being stressed. If there is indication of danger signs, the auditor would want to review if “corners were cut” at all and in what way.
Compliance within the rest of the bank
For each model, the Model Validation team comes up with a Model Summary detailing the following:
o Description of model
o Products that can be priced
o Model restrictions
o Stress testing results
o Model stability
This document is very useful for the bank. However in my experience, not everyone refers to and adheres to its restrictions and guidance. Important advice can be missed when this happens.
The auditor will want to review if there is a procedure in the bank to ensure adherence to this important document.
Model Risk
According to an interesting paper published in the Monetary and Economic Studies, 2000, “Model Risk and Its Control” by Toshiyasu Kato and Toshinao Yoshiba, model risk is defined as “the risk arising from the use of a model which cannot accurately evaluate market prices, or which is not a mainstream model in the market.” In risk measurement models, model risk is defined as “the risk of not accurately estimating the probability of future losses”.
Sources of model risk in pricing models can include use of wrong assumptions and errors in estimations of parameters. Sources of model risk in risk measurement models consist of the authenticity of the assumed distribution and errors in the logical framework of the model.
It is a normal practice for banks to set up a reserve to allow for this risk to accommodate for potential losses. Scenario analysis is undertaken for various fluctuation patterns of risk factors, to establish position limits.
The Model Validation team attempts to assess the model risk by conducting stress and back testing. The challenge in back testing is in obtaining the results from testing the pricing model in the past, i.e. generating data that was not actually available at that time. This task is inherently exposed to any inherent model risk. Thus the accuracy and reliability of the back testing results would be hampered, and it will be difficult to quantify the results.

The challenge in stress testing is including all the possible scenarios and adopting a realistic probability distribution. Looking back at the credit crisis, we have to wonder about the results when stress testing was done to the models for pricing and measuring the risk of credit debt obligations (CDOs).
The auditor will want to review in detail stress testing and back testing results to observe if the methodology is consistent and logical. He would also be interested to bounce some ideas with the risk manager to gauge how comfortable he is with the model testing results. Further, the auditor would also want to know the areas of improvement identified by the bank itself in reaction to the testing results. This gives him an idea of the extent of the model risk involved.
Difficulty with purchased models
When it comes to models purchased from external vendors, difficulty lies in obtaining enough detail on the application such as calibration results and future calibration needs, bugging results, stress testing and back testing results, if any were done.
The auditor would want to ensure that these important tests are performed adequately by at least one of the parties. If it is the task of the external vendor, the Model Validation team must obtain adequate proof in writing and must have avenues to perform continued testing in future. If the task is with the Model Validation team, the vendor must provide enough information to perform testing so as not to lose any important shortcoming in the model.
Conclusion
It is important to perceive the Model Validation team as modeling for insights, not numbers. The team cannot afford to get lost in the model and testing results and lose the big picture. As for the auditor, it is perhaps more effective if he reviews the reports produced by this team and his interviews with relevant parties to gain insight as to how much he can rely on the team’s opinion and reports.
As the Malaysian and Asian banks start introducing a wider spectrum of financial derivatives, I hope I have provided at least some of the basic tools you will need. Perhaps the points raised in this series
may not be as relevant to the current environment, but the best auditors, like scouts, must always be prepared. 

1 comment:

  1. Thanks for this info. Its very useful.
    Can you provide your thought on following pls ? -
    1. What is the MV team set-up used to be in Investment banks, e.g. what MV team size used to be per assets-class-wise (i.e. for OTC derivatives), what is the working procedure used to be for each team/member (e.g. Is one person validates one model end-2-end, or one person within team just does one part of it ...etc.). Interested to know the details of the MV procedure for each U/L assets, currency set-up, geographical set-up ...etc.
    Thanks in advance.
    shyammohan22@icloud.com;

    ReplyDelete