III AdvancedWeek 13 • Lesson 38Duration: 40 min

BEN Benchmark Comparison

Is your strategy actually better than simple alternatives?

Learning Objectives

•Learn how to properly benchmark a trading strategy
•Understand why beating random walks is necessary but not sufficient
•Compare strategies fairly using appropriate benchmarks

Explain Like I'm 5

You have to prove your strategy is better than dumb alternatives. "I made money" isn't enough — buy-and-hold also makes money. You need to show your strategy makes more money with less risk than simpler approaches. If you can't beat a coin flip, why are you trading?

Think of It This Way

Benchmarking is like comparing test scores. Saying "I scored 80%" means nothing without context. 80% when the class average is 40%? Excellent. 80% when the class average is 85%? Below average. Benchmarks provide the context.

1Essential Benchmarks

1. Random baseline. Random entry with the same risk management. If you can't beat random entries, your signal has no value. 2. Buy-and-hold. The simplest possible strategy. You need to beat this on a risk-adjusted basis (not necessarily in raw returns). 3. Naive rules. Simple moving average crossover, RSI threshold. If complex ML can't beat a 50/200 SMA cross, the ML isn't adding value. 4. Risk-free rate. Trading has opportunity cost. If you can't beat T-bills after accounting for time and stress, why bother? 5. Previous version. When improving a strategy, the old version is the benchmark. Did the "improvement" actually improve anything? A strong ML engine should: - Beat random entries by ~8-10% WR - Beat buy-and-hold on risk-adjusted returns (Sharpe 2x+ higher) - Beat SMA crossover by ~5-8% WR - Massively beat the risk-free rate

2The Random Baseline — Your First Hurdle

This one trips people up: randomly entering trades with proper risk management can actually be profitable. If you flip a coin for entries but use a 1:1.5 risk-reward ratio with proper position sizing, you'll break even or better — because the favorable R:R means even a 50% hit rate generates positive expectancy. So "my strategy makes money" is a low bar. Your strategy needs to beat random entries significantly: - Random with 1:1 R:R → ~50% WR (breakeven before costs) - Random with 1:1.5 R:R → ~50% WR but profitable due to R:R - Your strategy needs → 54%+ WR to prove the signal adds value The gap between your WR and the random WR is your "signal premium." A bigger gap means more confidence that your ML is doing something real. A small gap means most of your profit comes from position sizing and risk management — not prediction.

Strategy vs Benchmarks: Risk-Adjusted Returns

3Risk-Adjusted Comparison

Raw returns are misleading. Consider: - Strategy A returns 50% annually with 30% max drawdown - Strategy B returns 30% annually with 5% max drawdown Which is better? Strategy B — by a lot. Because you can lever B up to match A's returns and still have lower risk. The right metrics for comparison: - Sharpe ratio: return per unit of risk. Higher = better. - Sortino ratio: like Sharpe but only penalizes downside volatility. Better for asymmetric strategies. - Calmar ratio: return / max drawdown. Tells you the pain-to-gain ratio. - Information ratio: excess return over benchmark / tracking error. Measures consistency of outperformance. For prop firm trading, Calmar ratio is arguably the most important metric because max drawdown = account death. A strategy with Calmar > 3 is excellent.

4When Your Strategy Loses to a Benchmark

This happens more than people admit. You spend months building an ML pipeline and it can't beat a moving average crossover. What do you do? Option 1: Simplify the model. Maybe you're overcomplicating things. Try fewer features. Simpler architecture. Sometimes less is genuinely more. Option 2: Check your features. Are your features actually predictive? Run feature importance. If the top features are noise, the model is memorizing randomness. Option 3: Check the regime. Maybe ML adds value in some regimes but not others. Trending markets might favor simple trend-following. ML might shine in mixed or volatile regimes. Option 4: Accept it. Sometimes simple strategies genuinely beat complex ones. There's no shame in deploying a well-validated SMA crossover if it consistently outperforms your ML. Ego is expensive in trading. The willingness to throw away months of work when benchmarks say "this isn't good enough" is what separates profitable traders from stubborn ones.

Key Formulas

Information Ratio

Excess return over benchmark, divided by tracking error. Higher IR = better risk-adjusted outperformance. IR > 0.5 is good, > 1.0 is excellent.

Hands-On Code

Strategy Benchmark Comparison

python

import numpy as np

def benchmark_comparison(strategy_returns, benchmarks: dict):
    """Compare strategy against multiple benchmarks."""
    strat_sharpe = (strategy_returns.mean() / strategy_returns.std()
                    * np.sqrt(252))
    
    print(f"=== BENCHMARK COMPARISON ===")
    print(f"Strategy Sharpe: {strat_sharpe:.2f}")
    print(f"Strategy Total Return: "
          f"{np.prod(1 + strategy_returns) - 1:.1%}")
    print()
    
    for name, bench_returns in benchmarks.items():
        bench_sharpe = (bench_returns.mean() / bench_returns.std()
                        * np.sqrt(252))
        excess = strategy_returns - bench_returns
        ir = excess.mean() / excess.std() * np.sqrt(252)
        
        better = strat_sharpe > bench_sharpe
        print(f"vs {name}:")
        print(f"  Benchmark Sharpe: {bench_sharpe:.2f}")
        print(f"  Information Ratio: {ir:.2f}")
        print(f"  {'[PASS] Outperforms' if better else '[FAIL] Underperforms'}")
        print()

Benchmark every strategy against multiple alternatives. If you can't clearly beat simple benchmarks on a risk-adjusted basis, the complexity of ML isn't justified.

Knowledge Check

Q1.Your ML strategy returns 30% annually. Buy-and-hold returns 25%. Is your strategy better?

Assignment

Benchmark your strategy against: random entries, buy-and-hold, and a simple SMA crossover. Compute Information Ratio for each comparison. Does your strategy add value over all benchmarks?

← Previous Lesson Next Lesson →