Monitoring a trading bot: everything except P&L

Ask an operator what they watch and most will say the equity curve. It is the worst available health signal: slow, noisy, and confounded by the market. A bot can be silently broken for days inside normal-looking P&L, and a perfectly healthy bot can lose money for a week.

Separate the two questions

“Is the strategy working?” and “is the software working?” are different questions with different data. This article is about the second. The metrics below all share one property: you can tell whether they are wrong without knowing anything about the market.

The core set

Our candle pipeline exposes most of this as one health object per shard — connected sockets, subscribed topics, symbols, buffers, message count, corrections, reconnects, clock offset, and a boolean. In steady state that reads 10/10 shards, around 1 850 topics, 725 symbols, roughly 900 messages per second and an offset within about ±40 ms. Any one of those moving is visible immediately.

Reject rate by code, not in aggregate

An aggregate reject rate hides the interesting information. Break it down by venue error code, because each code means something different is broken:

• A rise in 110004 means the sizing logic is not accounting for margin correctly.
• A rise in 170137 means instrument filters have drifted from your cache.
• Any 10018 at all means an IP allowlist problem — usually a server move.
• Any 10005 at all means the API key permissions changed under you.
• A rise in 10006 means the request queue needs work, not more retries.

The error reference maps the common ones. What matters operationally is that codes which are normally zero should alert on their first occurrence, not on a threshold.

Alerts people will actually respond to

The failure mode of monitoring is alert fatigue. An operator who has learned to ignore the channel is worse off than one with no alerts, because they believe they are covered.

Three tiers works:

1. Page. Trading is happening that should not be, or is not happening when it should. Unresolvable order state. Kill switch fired. Position without a stop.
2. Notify. Degradation that has not caused loss yet: reject rate up, latency up, reconnect storm, queue growing.
3. Digest. Daily summary — fills, rejects by code, reconnects, worst latency, reconciliation deltas, limits that bound.

The digest is the one people skip and the one that catches slow degradation. A reject code that appears once a day for a week is invisible in real time and obvious in a weekly comparison.

Heartbeat the bot itself

A monitoring system that only reports when something is wrong cannot distinguish “everything is fine” from “the monitoring is dead”. Emit a positive heartbeat on a fixed interval and alert on its absence.

This is also how you get honest uptime numbers rather than remembered ones. We record equity snapshots on a fixed cadence for exactly this reason: gaps in the series are gaps in operation, and they are measurable rather than recalled. When we checked ours on a development rig, the honest figure was 86.4% with one 34-hour gap — which is a perfectly normal number for a rig under active development, and precisely why we do not print a 99.9% uptime claim on the site.

Log the negatives

Record why a signal did not fire, why an order was not placed, which risk rule refused. Our signal engine stores every evaluation with its factor values — 5.76 million of them over fifteen days — and skip reasons for each rejection.

It costs storage and it converts the most common operator question, “why is it doing nothing?”, from a debugging session into a query. That trade is worth making every time.