Backtesting Risk Models — The Whats, Whys and Hows
In the world of finance and investment, risk management is a critical component of decision-making. Investment professionals, portfolio managers, and financial institutions rely on risk models to quantify and manage the various forms of risk associated with their investments. These models aim to provide a comprehensive view of potential risks, helping stakeholders make informed decisions. However, like any other analytical tool, risk models are not perfect, and they require rigorous testing and validation to ensure their effectiveness. As history shows us, an error on the estimation of this probability can have critical consequences. An overestimation of risk leads to improper leverage of resources and a reduced competitiveness in markets that tend to be very disputed by players. An underestimation of risk can result in critical losses of capital, reputational damage and regulatory issues. One of the techniques to avoid such problems is backtesting the risk models, which we will explore in this article. The script developed to this article is in my Github.
What is a Risk Model?
A risk model is a mathematical framework or a set of statistical techniques used to assess and quantify the risks associated with a particular investment portfolio, asset, or financial instrument. These models can encompass a wide range of risk types, including market risk, credit risk, liquidity risk, and more. The primary goal of risk models is to provide insights into the potential losses or fluctuations in value that an investment may experience under different market conditions.
Common Types of Risk Model:
- Historical Models: Calculate risk from statistical metrics of the market behavior in the past.
- Stress Models: Assess how a portfolio would perform under extreme market conditions, often going beyond historical data.
- Monte Carlo Simulations: Utilize random variables to model various market scenarios and estimate portfolio risk.
Why Backtesting is Necessary
One thing that all risk models have in common is that they require assumptions and parameters to be developed. If these assumptions and parameters are not wisely chosen, even an excellent model will provide useless information. Backtesting is a crucial step in assessing the reliability and effectiveness of a risk model. This process involves comparing the model’s predictions with the actual outcomes observed during a historical period. By doing so, you can determine whether the risk model provides accurate estimates of risk and whether it can effectively guide investment decisions. Here are some reasons why backtesting is essential:
- Model Validation: Backtesting helps confirm whether the model accurately reflects the underlying risk factors, market dynamics, and financial instruments. It ensures that the model is a valid representation of reality and can be used in risk mitigation.
- Continuous Improvement: By comparing model predictions with actual outcomes, you can assess the model’s performance in various market conditions. This allows for improvements and adjustments to enhance accuracy.
- Regulatory Compliance: Many financial regulations and industry standards require institutions to backtest their risk models to ensure that they meet specific standards and thresholds and avoid systemic risk.
How to do it?
Let's do a quick study taking the USD/BRL currency pair as an example. The chart below presents its closing exchange rate since 2015 until November 2023.
From the prices we are able to calculate the daily returns and plot them in the chart below.
One of the most common approaches to model market risk is to calculate the Value at Risk (VaR) using an exponential weighted moving average (EWMA) of the returns. This method was proposed by the well known J.P. Morgan's and Reuters' RiskMetrics in 1996 and has been widely used ever since.
The volatility proposed by this model is a recursive function calculated as:
where sigma is the volatility, lambda is the decay factor and r is the asset return. The charts below present the effect of different decay factors in the weights of recent data points.
We know that it is a robust and valid model for many assets, but with so many other good options, how to find out if this is a good call for the USD/BRL pair? Also, as the study shows, the model decay factor parametrization is essential to its adequate predictions. A small decay factor will turn the model too responsive to short term data, and a large decay will make it not responsive at all, so how to define the appropriate value for this asset? This is where backtesting is helpful.
The Value at Risk (VaR) is calculated from the obtained volatility using the distribution Z-score based on a desired confidence level. For example, considering a two-tailed situation (where we can hold both long and short positions), the appropriate Z-score for a confidence level of 95% will be obtained from the 97.5% percentile, remaining 2.5% for each side of the returns distribution. That means that, if our model is carefully designed, there will be a 5% probability that the returns fall out of the estimated VaR. But using Z-scores implies in a fundamental assumption, that the distribution of the returns is normal. So before going forward, let's compare the USD/BRL returns distribution with a normal distribution in a histogram.
From a visual inspection they look very similar, so we will go one step further and apply a Shapiro-Wilk statistical test for normality.
The p-value presented is too low to assume normality of the distribution based on statistical significance. However, the obtained test statistic allow us to consider some practical significance and use z-scores in our VaR estimation. If you are looking for a more reliable method when the data is not normal, I suggest bootstrapping the original return distribution to calculate the confidence levels. We will calculate the EWMA applying a decay factor of 0.94 and use a 90% confidence level to estimate our VaR. The results are in the chart below.
As we can see the area between the VaR limits comprises most of the daily returns, and some extreme events fall out of the calculated ranges. To know if this model and the decay factor are adequate for this asset we must check if the occurrence of such events is equivalent to 10% as we defined in our confidence level. By calculating the percentage of events we can see that after the necessary stabilization period the estimation error is close to 10% in both expanding and rolling windows.
This is good news and shows a good validity of the EWMA VaR model to the USD/BRL pair. But why choosing such a complex model? What if we just take the historical return percentile to estimate our VaR? The picture below shows the estimated historical VaR using a 90% confidence level for a 3-year window. As we can see it is a much more static metric than the previous one.
Let's check the performance of both methods in the figure below
By looking at the results, we see that even after the necessary stabilization period the percentage error is different than the expected 10%. Both the expanding and rolling windows present values that oscillate much more than the EWMA error. In most windows, the simple historical VaR either overestimated or underestimated risk more than the EWMA VaR method.
We will stop here, but of course, this is just the beginning. In order to properly implement a risk model we must perform similar studies extensively to make sure that the future scenarios are covered as much as possible.
Conclusion
Backtesting risk models is a critical component of effective risk management in the financial industry. It helps us to verify and compare the reliability of different models in predicting and managing various forms of risk. By conducting thorough backtests, investment professionals and financial institutions can make more informed decisions and better protect their assets and investments. It’s essential to remain aware of the potential consequences of overestimation or underestimation of risk in risk models and use them as part of a broader risk management framework. Striking the right balance in risk estimation is crucial for achieving investment goals while mitigating potential losses.
Thank You!
Thank you for reading this material, I hope it can be useful for you in some way. If you want more details on the analysis steps you can check the script in my GitHub or reach me out in LinkedIn!
Subscribe to DDIntel Here.
Have a unique story to share? Submit to DDIntel here.
Join our creator ecosystem here.
DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.
DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1
