IV ExpertWeek 13 • Lesson 37Duration: 45 min

SEN Sensitivity Analysis

How fragile is your strategy? Parameter perturbation testing

Learning Objectives

•Understand sensitivity analysis and why it matters for strategy resilience
•Learn how to perturb parameters systematically
•Interpret sensitivity results for strategy confidence

Explain Like I'm 5

Sensitivity analysis asks: "If my parameters are slightly wrong, does everything fall apart?" If changing max_depth from 6 to 5 or 7 destroys your strategy, it's fragile and probably overfit. If it barely matters, the strategy is robust. You want insensitivity to small parameter changes.

Think of It This Way

Sensitivity analysis is like testing a recipe. If the cake is ruined by adding one extra gram of sugar, that's a fragile recipe. If it tastes great whether you use 95g or 105g, it's forgiving. Good strategies, like good recipes, aren't finicky about exact parameters.

1Parameter Sensitivity

For each important parameter, ask: "What happens if I change this by ±10-20%?" Parameters worth testing in a typical ML trading system: - Tree depth (5, 6, 7) - Number of estimators (150, 200, 250) - Learning rate (0.02, 0.03, 0.04) - Signal threshold (±0.02) - Risk per trade (±0.05%) - Drawdown zone boundaries (±1%) For each perturbation, run full walk-forward and compare: - Win rate change - Total R change - Max drawdown change - Breach probability change If win rate changes < 2% across all perturbations → strategy is robust ✅ If win rate changes > 5% for any perturbation → that parameter is critical and fragile ⚠️ The difference matters enormously. A strategy that's robust to parameter changes will survive the imprecision of live trading. A fragile strategy will not.

2Multi-Parameter Sensitivity

Single-parameter sensitivity misses interactions. Two parameters might be individually stable but jointly fragile. Grid search approach: test all combinations of parameter values. - max_depth × learning_rate → 3×3 = 9 combinations - Add n_estimators → 3×3×3 = 27 combinations Heatmap visualization: plot performance (color) against two parameters (axes). You want a broad "good" region (plateau), not a narrow peak. If the performance heatmap shows a sharp peak → strategy is fragile. One wrong setting and performance collapses. If it shows a broad plateau → strategy is resilient. Performance stays good across a range of settings. A well-validated system should show broad plateaus for most parameter pairs. That's evidence of genuine signal capture rather than parameter fitting.

3Sensitivity Heatmaps in Practice

Here's how you actually do this. Pick your two most important hyperparameters and create a grid. For a tree-based model, that's usually max_depth vs. learning_rate. Train a model at every combination and measure OOS win rate. What you want to see: the entire center of the heatmap is green (good performance). Edge cases at extreme values may degrade, but there's a big stable region. What you don't want: one tiny green cell surrounded by red. That means your performance depends on getting the exact right parameters — which you probably won't maintain in live trading conditions. This is one of the most underrated validation techniques. It takes maybe 2 hours of compute and gives you massive confidence — or rightful concern — about your parameter choices.

Parameter Sensitivity: Win Rate vs Max Depth

4When Sensitivity Analysis Saves You

Real talk: sensitivity analysis has saved me from deploying fragile strategies more than once. Pattern 1: Model looked great at learning_rate=0.03. At 0.025 or 0.035, performance dropped by 8%. That's a razor-thin edge that won't survive live execution noise. Killed it. Pattern 2: Model performed within 1-2% across the entire range of max_depth from 4 to 8. That broad plateau signals real signal capture. Deployed with confidence. Pattern 3: Model was stable for all parameters except the signal threshold. ±0.01 on threshold changed WR by 6%. Solution: use calibrated probabilities instead of a hard threshold. Fixed the fragility. The takeaway: if any single parameter makes or breaks your strategy, you don't have a strategy. You have a lucky parameter setting. And luck runs out.

Key Formulas

Sensitivity Coefficient

Normalized sensitivity: percentage change in performance per percentage change in parameter θᵢ. SC close to 0 = insensitive (good). SC ≫ 1 = highly sensitive (concerning).

Hands-On Code

Parameter Sensitivity Analysis

python

import numpy as np

def sensitivity_analysis(base_params, param_name, test_values,
                         run_backtest_fn):
    """Test strategy sensitivity to one parameter."""
    results = []
    for value in test_values:
        params = base_params.copy()
        params[param_name] = value
        perf = run_backtest_fn(params)
        results.append({'value': value, **perf})
    
    perfs = [r['win_rate'] for r in results]
    base_idx = len(test_values) // 2
    base_perf = perfs[base_idx]
    max_deviation = max(abs(p - base_perf) for p in perfs)
    
    print(f"=== SENSITIVITY: {param_name} ===")
    for r in results:
        bar = "█" * int(r['win_rate'] * 100)
        marker = " ← base" if r['value'] == base_params[param_name] else ""
        print(f"  {r['value']:>8}: {r['win_rate']:.1%} {bar}{marker}")
    
    print(f"Max deviation: {max_deviation:.2%}")
    print(f"  {'[PASS] Stable' if max_deviation < 0.02 else '[WARN] Sensitive'}")
    return results

Test each parameter at multiple values around the base. If performance barely changes, the strategy is resilient. Sharp changes indicate fragility — proceed with extreme caution.

Knowledge Check

Q1.Your strategy shows a broad performance plateau when varying max_depth from 4 to 8. What does this mean?

Assignment

Run sensitivity analysis for 3 key parameters of your strategy. Create heatmaps for 2-parameter combinations. Identify any fragile parameters and assess whether the strategy is stable enough for live deployment.

← Previous Lesson Next Lesson →