III AdvancedWeek 7 • Lesson 18Duration: 50 min

ENS Ensemble Methods

The wisdom of crowds — why committees beat individuals

Learning Objectives

•Understand why ensembles work (bias-variance decomposition)
•Learn the main ensemble techniques: bagging, boosting, stacking
•See how ensemble agreement serves as a confidence indicator
•Know when ensembles help and when they add complexity without benefit

Explain Like I'm 5

One analyst can be wrong. Five independent analysts are less likely to all be wrong in the same direction. Ensembles apply this logic to models: train multiple models, combine their predictions, and the result is more robust than any individual. The key word is "independent" — five copies of the same model are useless.

Think of It This Way

Think of jury duty. One person might have biases or make mistakes. Twelve people, each with different perspectives, are more likely to reach a fair verdict. But only if they truly deliberate independently — if they all just follow the loudest voice, the jury is no better than one person.

1Why Ensembles Work — The Math

The intuition is simple but the math makes it precise. Every model's error has two components: - Bias — Systematic error. The model consistently under- or over-predicts. - Variance — Random error. The model gives different answers on different training samples. Averaging multiple models: - Doesn't reduce bias — If all models are systematically wrong in the same way, the average is still wrong. - Does reduce variance — Random errors cancel out. Each model is wrong in a different direction, so the average is closer to truth. For this to work, models need to be diverse. Same model trained on the same data five times? Nearly identical errors. Same model trained on five different data subsets (bagging)? Different errors that cancel. Five different model types (stacking)? Even more different errors. In practice, a well-constructed ensemble reduces prediction variance by 20-40% compared to any single model. Breiman, L. (1996). "Bagging Predictors." Machine Learning, 24(2), 123-140.

Ensemble Error Reduction as Members Increase

2Bagging, Boosting, and Stacking

Bagging (Bootstrap Aggregating) Train the same model on different random subsets of the data. Average all predictions. Random Forest is bagging applied to decision trees. Each model sees a different slice of reality. Boosting Train models sequentially, with each new model focusing on the examples the previous one got wrong. XGBoost and LightGBM are both boosting methods. Models collaborate by correcting each other's mistakes. Stacking Train multiple different model types (XGBoost, random forest, logistic regression, neural net). Then train a "meta-model" that learns the optimal way to combine their predictions. Most powerful in theory, most complex in practice. For trading: - Bagging provides robustness and uncertainty estimates - Boosting maximizes prediction accuracy (what XGBoost already does) - Stacking works when you have genuinely diverse model types Most production systems use boosting (XGBoost/LightGBM) as the primary model, with a bagging layer on top for uncertainty estimation. Stacking is powerful but adds maintenance overhead. Dietterich, T.G. (2000). "Ensemble Methods in Machine Learning." MCS.

Performance by Ensemble Method (Out-of-Sample)

3Ensemble Agreement as Confidence

Here's where ensembles get really useful for trading: the agreement level among ensemble members is a natural confidence indicator. If you have 5 models and all 5 say "buy" — high confidence. If 3 say buy and 2 say sell — low confidence, maybe skip. This isn't fancy math. It's just counting votes. But it's surprisingly effective: • 5/5 agreement — High confidence. Full position size. • 4/5 agreement — Moderate confidence. Standard position size. • 3/5 agreement — Low confidence. Reduced size or skip. • < 3/5 agreement — No consensus. Skip. This naturally implements confidence-weighted position sizing without explicitly modeling confidence as a separate feature. The ensemble does both — predicts direction AND expresses certainty. Empirical observation: trades where all ensemble members agree have win rates 5-8 percentage points higher than trades where agreement is marginal. That's a significant edge.

Win Rate by Ensemble Agreement Level

4When Ensembles Don't Help

Ensembles aren't a free lunch. They add cost without benefit when: Models aren't diverse. Five XGBoost models with the same hyperparameters trained on slightly different subsets produce nearly identical predictions. You need genuine diversity — different model types, different feature sets, or different training objectives. The bottleneck is bias, not variance. If your features don't contain the signal, no amount of ensembling will fix that. Garbage in, averaged garbage out. Computational cost matters. In production, running 5 models instead of 1 takes 5x longer. For high-frequency applications, this latency might not be acceptable. Interpretability is critical. Explaining one model's decision is hard enough. Explaining an ensemble's aggregated decision is harder. Regulators and risk managers want to know why a trade was taken. The test: if adding a 5th ensemble member improves out-of-sample performance by less than 0.5%, it's not worth the complexity. Diminishing returns set in fast.

Key Formulas

Ensemble Prediction (Bagging)

Average prediction across M models. Each model f_m was trained on a different bootstrap sample. Variance decreases proportionally to 1/M for truly independent models.

Bias-Variance Decomposition

Total error is the sum of systematic error (bias), random error (variance), and irreducible noise. Ensembles reduce the variance component without affecting bias or noise.

Hands-On Code

Bootstrap Ensemble with Agreement Scoring

python

import numpy as np
import xgboost as xgb

class BootstrapEnsemble:
    """Bagged XGBoost ensemble with agreement-based confidence."""
    
    def __init__(self, n_models=5, **xgb_params):
        self.n_models = n_models
        self.params = xgb_params
        self.models = []
    
    def train(self, X, y):
        """Train ensemble on bootstrap samples."""
        n = len(X)
        for i in range(self.n_models):
            # Bootstrap sample (sample with replacement)
            idx = np.random.choice(n, size=n, replace=True)
            dtrain = xgb.DMatrix(X[idx], label=y[idx])
            model = xgb.train(self.params, dtrain, num_boost_round=300)
            self.models.append(model)
            print(f"  Model {i+1}/{self.n_models} trained")
    
    def predict(self, X):
        """Predict with agreement score."""
        dtest = xgb.DMatrix(X)
        preds = np.array([m.predict(dtest) for m in self.models])
        
        # Binary predictions per model
        votes = (preds > 0.5).astype(int)
        agreement = votes.mean(axis=0)  # % of models agreeing
        
        # Ensemble probability = mean of individual probabilities
        ensemble_prob = preds.mean(axis=0)
        
        return {
            'probability': ensemble_prob,
            'agreement': agreement,
            'confidence': np.abs(agreement - 0.5) * 2,  # 0-1 scale
        }

Five XGBoost models trained on different bootstrap samples. Agreement score acts as a natural confidence indicator. Full agreement = high confidence = full position size.

Knowledge Check

Q1.Why does averaging 5 model predictions reduce error?

Assignment

Build a 5-model bootstrap ensemble. For each test sample, record both the ensemble prediction and the agreement level. Plot win rate vs agreement level. Verify that high-agreement predictions have higher win rates.

← Previous Lesson Next Lesson →