Regime Switching Detection Using Hidden Markov Models

Regime Switching Detection Using Hidden Markov Models - Kyle E. Lamb

Introduction:

The financial markets are known to exhibit varying behavior over periods of days, weeks, months and even years. This likely due to investor sentiment, the condition of economic activity, political events, and government regulation. The ongoing challenge for investors is to identify the current condition of the market and act accordingly to put them in best position to see positive alpha in their portfolios. Most stockholders are aware of the terms bull and bear. They are terms that are meant to classify what type of market we are in currently and how it will behave in the future. Most investors will state that when stocks drop, we are in a bear market and when stocks go up, we are in a bull market. In doing so, the belief of such a classification will influence the decisions they make. The goal of this study is to design a regime detection investing model that will classify regimes based on a selection of mathematical data.

In a Nutshell:

This project will attempt to use Hidden Markov Models to classify the different market regimes that could exist, as well as their corresponding statistical performance. The goal of this project is to apply HMM as an investment strategy to ultimately increase the overall risk adjusted return on a portfolio.

Hidden Markov Models:

HMM is a generative model in which the system being modeled is considered to be a Markov process with a number of unobservable states. HMMs are meant to capture hidden information from observable time series data. The hidden information in our case is the specific regime or state that the market is in currently. An HMM indicates that each hidden state is only dependent on the previous state and the transition between states is based on the probability distribution defined during the training process of the model.

Data and Visualization:

The first step in designing a financial model is choosing appropriate data in which the model will be trained on. I decided to train the model on the SPY ETF which is meant to track the price movement of the S&P500 index. I chose the S&P500 because it is often considered the benchmark for overall economic health. Specifically for the observable variables, daily returns and volatility will be used to train the model, as well as a ten day moving average of the daily returns. For volatility, I will be using a ten day rolling standard deviation of the SPYs daily returns. The data that will be used for training our HMM can be seen below:

Model Training and Configuration:

For this study I used the Gaussian Hidden Markov model from the hmmlearn API in python for training and testing. For this model I will be attempting to create a two state HMM model for classifying bull and bear regimes. The main parameters of the model is number of hidden states, covariance type, and the max number of iterations. For hidden states, I chose two as I am only trying to classify bull and bear regimes. As for the covariance type, I chose to use a "full" covariance matrix due to the assumption that each state will be completely independent of each other. Using a full matrix ensures that each state will have its own covariance matrix as opposed to one matrix applying to all states. At the end of the study I will show a three state HMM to classify bull, bear, and kangaroo markets. We will look at the two state model first. As shown below, using the HMM model I classified approximately twenty years worth of S&P500 data into two discrete market regimes.

From the average daily returns and volatility from the two discrete hidden states classified by our model, we can see that regime 0, has relatively high returns and low volatility as compared to regime 1. The interpretation of each regime can be subjective. I chose to identify regime 0 as a bull regime as there is low volatility paired with high returns and regime 1 as a bear regime as there is high volatility and lower average daily returns. Ideally we would like to see the bear regime have negative returns as large drawdowns are typically associated with bear markets. However, due to the models limits, parts of the bear markets were classified as regime 0 which ultimately increased the overall average daily return.

Market Regime Breakdown

For testing, I created a back testing script in python. The specifications for the back test are as follows: when the model classifies regime 0 (bull regime) we will be 100% in the market, when the model classifies regime 1 (bear regime) we will be out of the market and investing into the 10 year treasury note as it is considered the risk free rate which will be a adequate baseline for basic hedging in the portfolio. This study is not meant to discuss hedging, there is likely a more efficient way to hedge against the market when returns are expected to be low. For the sake of the study, we are trying to increase the total risk adjusted returns on our portfolio, and investing in the 10 year treasury note is an effective way of achieving satisfactory results. The charts and tables below represent the strategy I created from the HMM model compared to the S&P500:

Full PDF Report (Training)

As seen from the performance report, we can conclude that the HMM model used on our in-sample data proved to be successful at increasing total cumulative return, as well as our overall risk adjusted return indicated by the Sharpe ratio. The model also decreased drawdown from 55.19% to 16.86% indicating that this strategy is effective at protecting against large market drops such as the 2000 dot com bubble and the 2008 financial crisis.

Model Testing and Validation:

To test our model, I have selected a portion of out-of-sample data to be used. For the testing process, the model will be trained on data from 2000-07-21 to 2017-01-03. To test, I classified regimes from 2017 to present time, day by day. Once again, if the model classifies a bull regime, we will be in the market, if a bear regime is classified, we will be investing in the 10 year treasury. Below is the HMM model after classifying approximately five years worth of data from 2017 to 2022:

Market Regime Breakdown

--------------------------------------------------------------------------------------------------------------------

From the regime breakdown, we can see that the performance within each regime was similar in daily returns. Volatility in regime 1 was 209% higher than regime 0. The model didn't perform as well on out-of-sample data compared to in-sample data, but according to the reports below its evident that using HMMs it was possible to increase the overall risk adjusted return over a five year period on the SPY.

Full PDF Report (Validation)

According to the full report, the model was able to completely avoid the volatility that came with Covid-19 recession, while maintaining similar returns throughout a 5 year period. We successfully increased the Sharpe ratio by 59% as well as keeping volatility and max drawdown significantly lower than the benchmarks

Although we did not beat the market throughout the period, we increased the overall risk adjusted returns which will allow us to incorporate several leveraging strategies in order to make up for that potential loss. A full pdf report regarding a simple 2x leveraged portfolio for trading the SPY based on our HMM model can be shown below.

Full PDF Report (2x leverage)

Conclusion:

As observed by the two performance reports from the testing phase, I concluded that HMMs are a viable strategy to increase Sharpe ratios and expected returns after adjusting for volatility. It is important to note, the model is solely dependent on the features used to train it on. There can be much further development on an HMM model compared to this article. Possible developments could be to introduce more observed features and increase the total number of hidden states to capture even more hidden information and create a much more versatile investing model. It is also important to note that when increasing the number of hidden states there will be more optimal strategies to employ within those states. My HMM model only includes two hidden states, and it is likely that investing in the 10 Year Treasury note is not an optimal strategy during the bear regime. Distinguishing the optimal strategies within each individual state is another challenge within itself. Yet, due to the models performance, I can conclude that this is an effective strategy and should be looked into deeper in the future.