If the operator cannot say why the bot entered, the bot should not be live

The argument for explainable models is usually framed around regulation or ethics. In trading there is a more immediate reason: an operator who does not understand why the system acted cannot decide whether to intervene.

The 4am problem

The bot has taken three losing trades in a row on the same instrument. You are awake, looking at a screen, and there are exactly two possibilities:

• The strategy is working as designed and this is a normal losing streak. Intervening destroys the edge you validated.
• Something is broken — a stale feed, a changed instrument spec, a regime the model has never seen. Not intervening compounds the damage.

Distinguishing these requires knowing what the model saw and why it acted. Without that, the decision is a coin flip dressed as judgement — and the predictable human response is to switch it off, which is wrong half the time.

What each decision should carry

Not a global feature-importance chart. A per-decision record, written at decision time:

@dataclass
class Decision:
    ts: datetime
    symbol: str
    action: str
    model_version: str          # which artefact produced this
    features: dict              # the exact values seen
    prediction: dict            # direction, expected MFE, expected MAE, confidence
    top_contributors: list      # 3-5 features that moved this prediction
    ev_calc: dict               # edge, cost, margin — the gate arithmetic
    risk_verdict: str           # approved / resized / rejected, and by which rule
    reason: str                 # human-readable, one line

The cost is storage and a few milliseconds. The benefit is that every question an operator can ask has an answer that does not require reconstructing state.

Record the refusals too

The most common operator question is not “why did it trade” but “why is it doing nothing”. That question is unanswerable unless rejections are logged with reasons.

Our signal engine stores every evaluation — 5.76 million over a fifteen-day window — with the factor values and the specific skip reason: series too short, doji in the sequence, expansion filter failed, RSI not extreme, impulse outside the corridor, no free slot, spread too wide. “Why no entries today” is a query.

This constrains model choice, and that is fine

Gradient boosting on named features gives you readable importance and per-sample attribution almost for free. A deep model over raw sequences does not, and post-hoc explanation methods produce plausible stories that are not always the actual mechanism.

Given that market prediction has low signal-to-noise anyway — where the accuracy gap between a boosted tree and a deep model is usually small — trading interpretability for a marginal metric improvement is a bad exchange. You lose the ability to operate the thing.

The test

Before a model goes live, one question: can the person on call explain, from the logs alone and within five minutes, why the system took its last three actions?

If not, the missing piece is not model quality. It is the record, and it is cheaper to add now than during the incident that will eventually require it.