The Curse of Dimensionality in Finance

In Big Data we often come across when we have more features than the sample data for our model, so we reduce the features using PCA or SVD or any other algorithm to reduce the features

Similarly in the field of Finance when one is making a Volatility model or let’s say making a portfolio using Efficient modern portfolio, we often come across with this curse Let’s take an Example

To obtain an Efficient Modern Portfolio of N assets one must estimate

1. N expected returns
2. N volatility
3. N(N-1)/2 (Here the problem arises)

Let’s say you want to select stocks for your portfolio from the BSE 100/FTSE100 index universe so we have 100 companies to choose from.

So, for our Efficient Portfolio, we have to estimate around

100 expected returns, 100 Volatility but around 4950 correlations parameters which is quite a lot while the sample size we have

1 year = 252 Trading Days

even we took 10 years of historical data our Estimates > Sample Data

Also,

As our universe increases the correlation parameter also increases quite a lot and demanding us a large sample size

If we have more features/parameters than our Sample Size it leads to overfitting our model as well as each of our parameter will contain a lot of noise and will not be very helpful

Also, more and more parameters lead to appear equally distant from each other in our risk-return sample space making Clustering Parameters harder and no meaningful clusters will be formed

Challenge - (How to reduce the dimensionality)

1.Easy approach: Just increase the sample size which can be done by increasing time-frequency or time history but we can do it at a certain limit and after that, we have to reduce our parameter

2.Constant Correlational Model

First, let’s understand what’s a sample covariance matrix

The covariance matrix is estimated via a traditional sample estimate.

let’s say Essentially if you need to estimate the covariance between stock i and stock j. You take a sample and you just measure the sample covariance, which is given by the average of the products return on stock i minus the mean return on that stock, times return on stock j minus the mean return on stock j, and then we take the average over the sample period of those gross products. That’s exactly the definition for covariance.

The constant correlation model imposes the assumption that all correlation parameters are identical. The reason we force them to be identical is that we want to impose some structure and reduce the number of parameters to estimate since it is better to estimate the single parameter accurately, than estimating many parameters

So, the objective is to cut down the correlation parameters from N(N-1)/2 to 1

The optimal Estimator of this constant correlation is

ρ = 1/(N(N-1))∑ρij

It’s bizarre that we are assuming that correlations between the stocks are the average correlations of our portfolio universe

But There’s been many studies where meaning portfolios come through using CCM models

So, in the end for our portfolio from the BSE 100/FTSE100 index universe we manage to reduce (100 expected returns, 100 Volatility, and around 4950 correlations parameters) to (100 expected returns, 100 Volatility, and 1 correlations parameters)

3.Estimating Covariance matrix using Factor Models

Here’s a typical Factor decomposition model looks like

Ri,t = αi+ β(i,k)F(k,t) + εi,t

where β(i,k) is the sensitivity of stock I with the Factor K

Now using the Factor model, we can get the expression in terms of

Variance :

Co Variance

Now let’s start imposing structure that specific risk on the two stock I and J are uncorrelated such that COV(epsilon i, epsilon j) =0 in order to reduce the estimated parameters,

By doing that we are introducing model risk but we manage to reduce the correlation parameter by a factor of 10 for our universe

we have 100 stocks and say 2 factors,

then we need to estimate Betas for each stock with respect to each factor.

We first need 100 volatility estimates for individual stock returns, plus 100 estimates of betas of stocks with respect to factor 1, 100 estimates of betas of stocks with respect to factor 2, and finally 2 volatility estimates for factor returns, which gives a total of 100+100+100+2=302

4. Another Approach is by Shrinking Covariance Matrix

Honey, I Shrunk the Sample Covariance Matrix by Olivier Ledoit, Michael Wolf :: SSRN

So, in the above methodologies we discussed, we reduced the sample risk by introducing model risk via imposing conditions.

Methodologies like the constant correlation methodology or the factor base methodology that suffer from a lower degree of sample risk, because we reduce the number of parameters to estimate.

But that came at the cost of introducing some kind of structure/assumption, and therefore there is a fair amount of model risk.

The idea behind the shrinkage method is that, we’re not going to have to choose either to go for higher sample risk or higher model risk, we are going to take two methodologies and we’re going to mix them

Statistical shrinkage estimators are mathematically designed to deliver the optimal trade-off between model risk and sample risk.

It is based on two covariance estimate matrix one with High sample risk and another one with high model risk

Using methodology introduced in this paper Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps | NBER

Shrinkage can be done by imposing Weight's constraints. Introducing Min/Max weight Constraints will help us to get reasonable portfolios

Now we try to implement the above methodologies in Python and see our portfolio returns :

I randomly selected 35 stocks from y finance

Now for our portfolio, we are going to write an optimizer again for our weighting scheme which we wrote in our previous posts at the end where we minimize the volatility similarly, we are going to maximize the Sharpe ratio by minimizing the negative Sharpe ratio as these model weights are not dependent on the expected returns instead, they are dependent on the correlation of the stocks

We will make # different GMV portfolio where weights will depend on

1. Sample Correlation
2. Constant Correlation
3. Shrinkage between sample risk and model risk

As you can see our GMV Shrinkage ends up in between the GMV sample and GMV CC