Look-ahead bias: how an automated audit found three real leaks in our own model

A look-ahead leak is any path by which information from time t+1 reaches a decision made at time t. In a backtest this is invisible and flattering: metrics improve, the equity curve smooths, and every incentive you have points away from investigating.

Where leaks actually come from

Not from anything exotic. The recurring sources are mundane:

• Off-by-one in alignment. Joining open interest, funding or a higher-timeframe series onto your bars and taking index -1 — which is the current, not-yet-complete value. This is the single most common one.
• Using the closing bar you are trading on. If the signal uses the close of bar t, you cannot enter at the open of bar t.
• Normalisation over the whole dataset. Computing a mean, standard deviation or scaler on all the data, then splitting into train and test. The test-set statistics are now in the training features.
• Labels that peek. A target defined as “maximum move over the next hour” is fine; a feature computed from the same window is not.
• Survivorship in the universe. Selecting today's liquid symbols and testing them over history is a bet on knowing which ones would survive.
• Resampling that fills forward across the boundary. Reindexing a slower series onto a faster one with a forward-fill that starts before the value existed.

The audit that actually works

Reading code for leaks does not scale and does not find alignment bugs. The method that does is mechanical: recompute every feature at bar t on a series that has been truncated at t, so the future does not physically exist, and compare against the value your normal pipeline produced.

Any mismatch is a leak, by definition. There is no interpretation step and no judgement call.

def audit(df, feature_fn, sample_points, tol=1e-9):
    """Recompute each feature on a truncated series. Any difference is a leak."""
    full = feature_fn(df)
    leaks = []
    for t in sample_points:
        truncated = df.iloc[: t + 1]          # the future does not exist here
        partial = feature_fn(truncated)
        for col in full.columns:
            a, b = full[col].iloc[t], partial[col].iloc[-1]
            if not close_enough(a, b, tol):
                leaks.append((col, t, a, b))
    return leaks

Run it over a few thousand random bars across different regimes, and run it in CI so a leak cannot be reintroduced by a later change.

What ours found

We built this audit for our pump-fade signal model — 59 features across episode structure, higher timeframe context, the BTC background and liquidity. It found three genuine leaks, and all three were alignment bugs rather than conceptual errors.

The pattern was the same each time: when open interest, funding and the BTC background were aligned to the bar series by timestamp, index -1 referred to a value that would not be known until after the decision point. Every one of them made the model look better. None of them would have been found by reading the code, because the code looked correct — the join was written the way such joins are usually written.

Purge and embargo

Cross-validation designed for independent rows is wrong for time series. If your label looks forward one hour, then a training sample immediately before a validation sample shares that hour — the split leaks across its own boundary.

The fix is a gap sized to the label horizon: purge the training samples whose label windows overlap the validation set, and embargo a further margin after it to account for autocorrelation. This is not optional refinement; a random split on time-ordered data will beat an honest one every single time, and the difference is entirely fictional.

We built purging and embargo in as defaults rather than options in TideGBM, along with a strict mode that refuses to fit on features failing the look-ahead lint. Making the honest path the default is the only version of this that survives a deadline.

Then publish the honest number

After closing the three leaks, the standalone out-of-sample AUC of our three geometric formulas sits at roughly 0.50–0.52. That is barely better than a coin flip, and we say so on the case study page.

The predictive power in that model comes from elsewhere — episode structure, how many legs the move has already run, how long it has lasted, the depth of its worst pullback, the instrument's liquidity, and BTC volatility. The formulas earn their place as an independent cross-check and as the explanation shown to an operator. Presenting them as a source of profit would not be true.

A short protocol

1. Write the audit before the model. It is a hundred lines and it changes what you build.
2. Sample thousands of bars across regimes, not a handful.
3. Purge and embargo by default in every split.
4. Run the audit in CI. Leaks come back with the next refactor.
5. Publish the honest out-of-sample number, including when it is unimpressive.

The last one is the hardest and the most useful. A model whose weaknesses are documented can be improved. A model whose backtest is beautiful for reasons nobody has checked is a liability wearing a nice curve.