Walk-forward validation, done properly

The single most common methodological error in strategy research is applying cross-validation designed for independent samples to data that is anything but. It is easy to fix and it changes results dramatically — downward, which is why it gets skipped.

Why a random split leaks

Two reasons, both sufficient on their own.

Label overlap. If your label looks forward one hour, a training sample at 10:00 has a label covering until 11:00. A validation sample at 10:30 shares that window. The model has seen the answer.

Autocorrelation. Adjacent observations are near-duplicates. A random split puts near-identical rows in both train and validation, so the model is being tested on data it effectively memorised.

The structure

Train on a window, validate on the period immediately after it, then move forward. Between them, a gap:

def walk_forward(index, train_span, test_span, horizon, embargo):
    """Purge the label horizon, then embargo for autocorrelation."""
    start = 0
    while start + train_span + horizon + embargo + test_span <= len(index):
        train = slice(start, start + train_span)
        gap   = horizon + embargo          # nothing from here enters training
        test  = slice(start + train_span + gap,
                      start + train_span + gap + test_span)
        yield train, test
        start += test_span                 # non-overlapping test periods

Purge is sized by the label horizon: remove any training sample whose label window reaches into the test period. Embargo is an additional margin for autocorrelation, typically a small multiple of the horizon.

Anchored or rolling

For crypto we usually prefer rolling, because market structure changes fast enough that three-year-old microstructure is a different market. Run both if you can — a strategy that only works anchored is telling you something about how much history it needs.

How many folds

Enough that a single lucky period cannot carry the result. Under ten folds, one good quarter dominates. The constraint is that each test period must contain enough trades to mean anything — a fold with four trades is noise.

The arithmetic: total trades divided by folds should leave at least 30 per fold, preferably more. If it does not, you either need more history, a higher-frequency strategy, or the honest conclusion that this cannot be validated yet — see when not to automate.

What to report

Not the average across folds. The distribution:

• Result per fold, listed. Consistency matters more than the mean.
• How many folds were profitable. Eight of ten is a different claim from five of ten with two large winners.
• Worst fold. This is your realistic bad quarter.
• Whether performance trends downward across folds — that is decay, and it is the most important thing in the table. See alpha decay.
• Parameter stability: did the optimum move between folds? A wandering optimum means you are fitting noise.

The discipline that makes it work

Walk-forward only means anything if you do not iterate against it. If you tune parameters, look at the walk-forward result, tune again, and repeat, you have turned your validation set into a training set through your own decisions.

Hold out a final period you look at exactly once, at the end, and treat that number as the result. It is the only number in the whole process that has not been contaminated by your choices — which is why it is usually the disappointing one.