Model drift: knowing that the strategy stopped working before the account does

There are two ways to find out that a strategy has stopped working. You can watch the equity curve, which will tell you in six weeks with poor confidence. Or you can watch the machinery, which will tell you in days.

Three kinds of drift, with different fixes

Distinguishing them matters because the responses are opposite. Retraining on input drift is unnecessary churn. Not retraining on concept drift is trading a model that has been invalidated.

What to instrument

• Feature distributions. Per feature, a rolling comparison against the training distribution. A population stability index or a simple quantile comparison is enough; the point is that it exists and alarms.
• Prediction distribution. If the model used to fire on 2% of evaluations and now fires on 9%, something changed before any trade result told you.
• Calibration. Bucket predictions by confidence and compare predicted against realised rate. A model can keep its ranking ability while losing calibration, and position sizing depends on calibration.
• Realised versus expected edge. Predicted 12 bps, captured 4. Split into slippage and decay so you know which one to fix.
• Rolling signal correlation. The single most useful number. It moves before P&L and it is easy to compute.

# Runs nightly. Cheap, and it moves weeks before the equity curve does.
def drift_report(recent, baseline):
    return {
        "psi":         {f: psi(recent[f], baseline[f]) for f in FEATURES},
        "pred_shift":  ks_stat(recent.pred, baseline.pred),
        "calibration": brier(recent.pred, recent.outcome) - BASELINE_BRIER,
        "signal_corr": spearman(recent.pred, recent.outcome),   # watch this one
    }

Write the shutdown rule before you deploy

This is the part that requires discipline rather than engineering. A rule decided while the strategy is losing money is not the same rule you would have written beforehand — the losing version always includes “give it another week”.

A workable form:

1. Reduce size when rolling signal correlation falls below a threshold for two consecutive weeks.
2. Stop opening new positions if it stays below for four.
3. Full shutdown and review on breach of the drawdown limit, regardless of what the diagnostics say.
4. Restart only after a fresh out-of-sample validation, not after the curve “looks better”.

Retraining is not automatically the answer

Scheduled retraining feels responsible and quietly introduces two problems. It makes the deployed model a moving target, so you can never fully attribute a result. And it hides concept drift: the model keeps fitting recent data and keeps reporting reasonable in-sample metrics while the edge underneath is gone.

Better: retrain on a fixed schedule for freshness, but validate every retrained model against a held-out period the same way you validated the first one. If the retrained model fails that bar, the answer is not another retrain — the answer is that the strategy is finished, which is a normal outcome and worth planning for. See alpha decay.

The cheapest version

If you implement one thing: log every prediction with its realised outcome, and compute the rolling correlation weekly. It is a few lines, it costs nothing, and it is the difference between noticing a dying strategy in two weeks and noticing it in two months.