IV ExpertWeek 16 • Lesson 49Duration: 45 min

RSM Regime-Switching Models

Formal frameworks for markets that change their mind

Learning Objectives

•Understand Hidden Markov Models (HMMs) for financial regime detection
•Implement and interpret transition matrices
•Compare HMMs to simpler regime identification methods
•Build a practical regime-switching strategy

Explain Like I'm 5

A Hidden Markov Model assumes the market is always in one of a few hidden states, and you can't see the state directly — you can only infer it from what the market does.

Think of It This Way

Imagine you can hear someone in the next room but can't see them. By listening to the sounds they make, you infer what they're doing. HMMs do the same with market observables — you hear the returns, but the regime is hidden.

1The HMM Framework

A Hidden Markov Model for financial markets has three components: 1. Hidden states. The unobservable market regimes (typically 2-4). Common specifications: - 2 states: Bull / Bear - 3 states: Low Vol / Normal / Crisis - 4 states: Quiet Trend / Volatile Trend / Choppy / Crisis 2. Emission distributions. Each state generates observable returns from a different distribution:

r_t | S_t = k \sim N(\mu_k, \sigma_k^2)

State 1 might have

\mu_1 = +0.05\%

daily with

\sigma_1 = 0.5\%

, while State 3 has

\mu_3 = -0.10\%

with

\sigma_3 = 2.0\%

. 3. Transition matrix. The probability of moving from one state to another:

P(S_t = j | S_{t-1} = i) = A_{ij}

A typical transition matrix for a 3-state model: | From / To | Low Vol | Normal | Crisis | |-----------|---------|--------|--------| | Low Vol | 0.95 | 0.04 | 0.01 | | Normal | 0.05 | 0.90 | 0.05 | | Crisis | 0.02 | 0.08 | 0.90 | Notice the high diagonal values — regimes are "sticky." This is both a strength (regimes persist) and a weakness (the model is slow to detect transitions).

2Transition Matrix Insights

The transition matrix is arguably the most useful output of an HMM for trading purposes. Expected regime duration: If the self-transition probability is

p_{ii}

, the expected duration in state

i

is:

E[T_i] = \frac{1}{1 - p_{ii}}

So a state with

p_{ii} = 0.95

has an expected duration of 20 periods. This tells you how frequently to expect regime switches. Transition probabilities to dangerous states: The off-diagonal elements tell you the probability of moving to a crisis state from your current state. Direct implications for tail risk management. In practice, you want different levels of stickiness for different decisions: - Position sizing: Use a sticky model (

p_{ii}

> 0.95) - Signal selection: Moderate stickiness (

p_{ii}

around 0.90) - Risk limits: Responsive model (

p_{ii}

around 0.85) — better to be conservative too early than too late

3HMM vs. Simple Methods

The quant community has a complicated relationship with HMMs. They're theoretically elegant but practically frustrating. Here's a candid comparison: | Criterion | HMM | Simple Indicators (Hurst, ADX, Vol) | |-----------|-----|-------------------------------------| | Theoretical foundation | Strong | Heuristic | | Interpretability | Moderate | High | | Estimation stability | Poor | Good | | Out-of-sample performance | Mixed | Decent | | Computational cost | High | Low | | Parameter sensitivity | Very sensitive | Moderate | The uncomfortable truth: in my experience, simple regime indicators (rolling Hurst + rolling volatility + ADX) matched or outperformed HMMs in most practical applications. HMMs shine when you need probabilistic state estimates or when the emission distributions are genuinely multi-modal. Bulla & Bulla (2006) showed HMMs improve asset allocation in some contexts, while Nystrup et al. (2015) found that the advantage disappears when you account for estimation uncertainty and transaction costs from regime switches.

4Building a Regime-Switching Strategy

If you want to incorporate regime-switching into a live system, here's a practical approach that avoids the worst HMM pitfalls: Step 1: Define regimes using simple indicators. Don't start with HMMs. Use rolling Hurst, realized vol percentile, and ADX to define 3-4 market states with clear boundaries. Step 2: Compute regime-conditional performance of your signals. For each signal, compute IC and Sharpe within each regime. This tells you which signals to activate in which states. Step 3: Build a smooth transition function. Don't use binary on/off switches. Use a sigmoid function:

w_{signal}(x) = \frac{1}{1 + e^{-k(x - x_0)}}

Where

x

is the regime indicator value,

x_0

is the threshold, and

k

controls transition sharpness. Step 4: Apply position sizing decay. When transitioning between regimes, reduce position size during the transition period to protect against whipsaw.

5Practical Regime-Switching Insights

After building and running regime-switching systems, here are the lessons that matter: 1. Fewer states is almost always better. 2-3 states capture 80%+ of the value. 2. Regime detection should be slow. A detector that switches daily is too noisy. Target weekly or monthly regime updates. 3. Focus on the dangerous regimes. If you can reliably identify regime transitions into high-volatility or crisis states, that alone is valuable. 4. Use regime detection for risk, not just alpha. Regime-conditional risk management (scaling position size by volatility regime, widening stops in crisis) reliably improves risk-adjusted returns. 5. Monitor detector stability. If your regime classification for a given historical period changes significantly when you add one more month of data, the model is unstable.

Key Formulas

HMM Transition Matrix

Probability of transitioning from state i to state j, with rows summing to 1

Expected Regime Duration

Expected number of periods in state i given self-transition probability p_ii

Hands-On Code

Simple Regime Detection (No Library Required)

python

import numpy as np

def rolling_regime_detector(returns, vol_window=60, n_regimes=3):
    """Simple regime detection using rolling stats + percentile bins."""
    n = len(returns)
    
    rolling_vol = np.array([
        np.std(returns[max(0,i-vol_window):i]) * np.sqrt(252)
        for i in range(vol_window, n)
    ])
    
    # Classify using percentile-based boundaries
    vol_pctile = np.percentile(rolling_vol, [33, 67])
    
    regimes = []
    for i in range(len(rolling_vol)):
        vol = rolling_vol[i]
        if vol <= vol_pctile[0]:
            regime = 0  # Low volatility
        elif vol >= vol_pctile[1]:
            regime = 2  # High volatility
        else:
            regime = 1  # Normal
        regimes.append(regime)
    
    regimes = np.array(regimes)
    
    # Compute transition matrix
    transitions = np.zeros((n_regimes, n_regimes))
    for i in range(len(regimes) - 1):
        transitions[regimes[i], regimes[i+1]] += 1
    
    row_sums = transitions.sum(axis=1, keepdims=True)
    transition_matrix = transitions / np.maximum(row_sums, 1)
    
    return {
        'current_regime': int(regimes[-1]),
        'regime_labels': ['low_vol', 'normal', 'high_vol'],
        'transition_matrix': transition_matrix.round(3).tolist(),
        'expected_durations': [
            round(1 / (1 - transition_matrix[i,i]), 1)
            for i in range(n_regimes)
        ]
    }

Implements regime detection using rolling volatility and return statistics with percentile-based boundaries, computing a transition matrix and expected regime durations.

Knowledge Check

Q1.In an HMM with self-transition probability p_ii = 0.95, the expected duration in state i is:

Q2.What is the primary practical advantage of simple indicators (Hurst, ADX) over HMMs?

Assignment

Implement a 2-state regime detector for EUR/USD using (a) a rolling Hurst + volatility approach and (b) a simple HMM (e.g., using hmmlearn). Compare the regime classifications — when do they agree? When they disagree, which was more predictive of forward 20-day returns?

← Previous Lesson Next Lesson →