← Back to Learn
IV ExpertWeek 19 • Lesson 56Duration: 55 min

KF Kalman Filtering

Adaptive hedge ratios — because static betas are indefensible in production

Learning Objectives

  • •Understand the Kalman filter conceptually and mathematically
  • •Implement a Kalman filter for dynamic hedge ratio estimation
  • •Compare static vs. dynamic hedge ratios in practice

Explain Like I'm 5

The hedge ratio between two assets isn't fixed — it drifts over time. A static hedge ratio from OLS gets stale fast. The Kalman filter continuously updates the hedge ratio as new data arrives, adapting to changing relationships. Think of it as GPS — constantly recalculating your position based on new information.

Think of It This Way

Imagine tracking a moving target with a rifle scope. OLS regression is like taking a photo and aiming where the target WAS 5 seconds ago. Kalman filtering is like using a tracking scope that continuously adjusts your aim as the target moves. The accuracy difference is not subtle.

1Why Static Hedge Ratios Fail

Four reasons why OLS hedge ratios will eventually betray you: 1. They're backward-looking by definition. Computed on historical data that may no longer describe current market dynamics. 2. The true relationship changes over time. Regime shifts, central bank decisions, macro shocks — all alter the covariance structure between assets. 3. Stale hedge ratios create phantom directional exposure. Your "market-neutral" position isn't actually neutral anymore. 4. This creates risk you didn't sign up for. You think you're running a spread trade, but you're actually carrying a significant directional bet. I've seen traders run "neutral" pairs trades that were actually carrying massive directional risk because their hedge ratio was computed 3 months ago and the relationship had shifted. That's not a pairs trade anymore — it's just two unrelated positions. The Kalman filter solves this by treating the hedge ratio as a hidden state that evolves over time. It gives you: - Optimal estimate of the current hedge ratio (not the historical average) - Uncertainty around that estimate (so you know when to trust it less) - Automatic adaptation to changing conditions (zero manual intervention)

2Kalman Filter Mechanics

The Kalman filter sounds intimidating but it's actually just two steps alternating forever: Predict step: Estimate the hedge ratio at time tt based on time t−1t-1: - βpredict=βt−1\beta_{predict} = \beta_{t-1} (assume hedge ratio persists) - Ppredict=Pt−1+QP_{predict} = P_{t-1} + Q (uncertainty grows over time because things change) Update step: Correct the estimate using the new observation: - K=PpredictPpredict+RK = \frac{P_{predict}}{P_{predict} + R} — this is the Kalman gain - βt=βpredict+K×(observation−prediction)\beta_t = \beta_{predict} + K \times (\text{observation} - \text{prediction}) — corrected estimate - Pt=(1−K)×PpredictP_t = (1 - K) \times P_{predict} — uncertainty decreases because you learned something That's it. Predict, observe, correct, repeat. Key parameters: - Q (process noise): How much β\beta changes per period. Higher Q = more adaptive, noisier. Typically 10−510^{-5} to 10−310^{-3}. - R (observation noise): Measurement noise. Typically estimated from residual variance.

3Static vs. Dynamic Hedge Ratio

The chart below compares a static OLS hedge ratio (computed once on historical data) versus a Kalman-filtered hedge ratio that adapts in real-time. Notice how the static ratio just sits there while the true relationship drifts around. The Kalman filter tracks the changes, staying close to the actual value. Why this matters for your spread: - Static ratio produces a spread with artificial trends (non-stationary artifacts) - Kalman ratio produces a cleaner, more stationary spread with better z-scores - Cleaner spread means more reliable entry/exit signals, which means better PnL The improvement isn't theoretical — Kalman-filtered spreads typically pass stationarity tests when OLS spreads fail. That's a direct, measurable upgrade to your signal quality.

Static OLS vs Kalman-Filtered Hedge Ratio

4Tuning Q: The Key Decision

Tuning Q (process noise) is the single most important decision with the Kalman filter. Here's the tradeoff: Q too low (e.g., 10−610^{-6}): - Filter barely adapts - Hedge ratio stays almost static - You miss real changes in the relationship - Basically just OLS with extra computational overhead Q too high (e.g., 10−210^{-2}): - Filter overreacts to every tick - Hedge ratio is noisy and jumpy - Your spread looks like random noise - Impossible to generate meaningful trading signals Q just right (typically 10−410^{-4} to 10−310^{-3}): - Adapts to real regime shifts over days/weeks - Ignores intrabar noise - Produces smooth, stationary spread How to find the right Q: cross-validate. Test multiple values of Q, compute the ADF test statistic on the resulting spread for each. The Q that produces the most stationary spread wins. It's that straightforward.

5When Kalman Is Not Worth It

Honest assessment — Kalman filtering isn't always the right answer: - If your pair is rock-solid cointegrated with stable β\beta, OLS is fine. Kalman adds complexity for minimal gain. - If you're trading on daily+ timeframes, the hedge ratio changes so slowly that monthly re-estimation with OLS works. - If you can't tune Q properly, a bad Kalman filter is worse than OLS. Kalman really shines when: - The relationship shifts over months (macro regime changes) - You're trading on intraday timeframes where freshness matters - You need the uncertainty estimate PP for position sizing For most retail stat arb, start with OLS. If you see the spread trending (OLS residuals failing ADF), then upgrade to Kalman. Don't over-engineer from day one.

Key Formulas

Kalman Gain

K determines how much to trust new data vs. the old estimate. High K means fast adaptation but noisy estimates. Low K means smooth but slow.

State Update

The hedge ratio estimate is corrected by the Kalman gain times the prediction error, continuously adapting beta to new data.

Hands-On Code

Kalman Filter for Dynamic Hedge Ratio

python
import numpy as np

class KalmanHedgeRatio:
    """Kalman filter for dynamic hedge ratio estimation."""
    
    def __init__(self, Q=1e-4, R=1.0):
        self.Q = Q  # process noise
        self.R = R  # observation noise
        self.beta = 0.0  # initial hedge ratio
        self.P = 1.0  # initial uncertainty
    
    def update(self, x, y):
        """Update hedge ratio with new observation."""
        # Predict
        beta_pred = self.beta
        P_pred = self.P + self.Q
        
        # Update
        innovation = y - x * beta_pred
        S = x * P_pred * x + self.R
        K = P_pred * x / S  # Kalman gain
        
        self.beta = beta_pred + K * innovation
        self.P = (1 - K * x) * P_pred
        
        return self.beta, self.P
    
    def run(self, price_a, price_b):
        """Run Kalman filter on price series."""
        betas = []
        spreads = []
        
        for a, b in zip(price_a, price_b):
            beta, P = self.update(np.log(b), np.log(a))
            betas.append(beta)
            spread = np.log(a) - beta * np.log(b)
            spreads.append(spread)
        
        spreads = np.array(spreads)
        betas = np.array(betas)
        
        # Z-score of spread
        lookback = 50
        z = (spreads - np.mean(spreads[-lookback:])) / np.std(spreads[-lookback:])
        
        print(f"Current beta: {betas[-1]:.4f}")
        print(f"Current z: {z[-1]:.2f}")
        
        return betas, z

Implements a Kalman filter that continuously adapts the hedge ratio between two price series, producing more stationary spreads and more reliable z-scores for trading signals.

Knowledge Check

Q1.The Kalman gain K is very high. What does this mean?

Assignment

Implement a Kalman filter for the hedge ratio between a cointegrated pair. Compare the Kalman-filtered spread's stationarity (ADF test) vs. the OLS spread. Which produces better pairs trading signals?

← Previous LessonNext Lesson →