Reinforcement learning for trading: an honest assessment

Reinforcement learning is genuinely the right shape for trading. An agent takes actions in an environment, receives delayed rewards, and must balance exploration against exploitation. That framing is correct, which is why the idea keeps coming back.

The problems are not conceptual. They are four practical obstacles that most attempts do not survive.

1. RL learns the simulator, not the market

RL needs to try actions and observe consequences, millions of times. You cannot do that live, so you train against a simulator — and the agent will exploit every flaw in that simulator with extraordinary efficiency.

If your simulator fills limit orders whenever price touches the level, the agent learns to place limits at implausible levels. If it ignores queue position, the agent learns to be a market maker with a perfect fill rate. The resulting policy is optimal against your code and meaningless against a venue.

2. Market impact is the part you cannot simulate

In a replay of historical data, your orders do not affect price. In reality they do: you consume depth, you signal intent, other participants react. An agent trained without impact learns strategies that assume infinite liquidity at the displayed price.

Modelling impact well is a research problem in its own right, and doing it badly is worse than not doing it — you get confidence in numbers built on a guess.

3. Reward design is where the strategy actually gets decided

Naive reward is P&L, and it produces an agent that takes enormous risk for marginal return, because unbounded leverage maximises expected P&L in a simulator with no ruin.

Risk-adjusted rewards help and introduce their own distortions: Sharpe over an episode encourages minimising activity; drawdown penalties encourage holding losers rather than realising them. Every reward function is a strategy specification, and it is a harder one to write than the strategy you were trying to avoid writing.

4. Sample efficiency versus available history

Modern RL is famously sample-hungry — the impressive results come from environments where you can generate unlimited episodes. Financial history is fixed and short, and each regime appears a handful of times.

Training on synthetic data helps with volume and hurts with fidelity: the agent learns the generator's assumptions. Training on replay repeatedly overfits the specific path history took.

Where it does fit: execution

There is one trading problem where RL is a genuinely good match, and it is not the one people ask for. Optimal execution — given a decision to buy X over the next N minutes, how do you slice it — has the properties RL needs:

• The action space is small and well defined: how aggressive to be, right now.
• Feedback is fast and measurable: implementation shortfall against arrival price.
• The simulator is more tractable, because you are modelling your own impact on a short horizon rather than predicting the market.
• The counterfactual is available: you can compare against TWAP and other baselines.

This is unglamorous and it is where the value is. Execution improvements compound across every trade a strategy makes, and unlike a signal, they do not decay when someone else discovers them.

If you want to try it anyway

1. Build the simulator first and validate it independently: replay historical fills and check your simulator reproduces them.
2. Include impact, even crudely. A wrong impact model beats none.
3. Start with execution, not with direction.
4. Validate the learned policy the same way you would validate any model — purged, out of sample, costs inside.
5. Compare against a simple baseline. If TWAP wins, that is the answer.

Our own position: we will build this if a client wants it, and we will say up front that a boosted tree on good features plus a solid execution layer usually reaches production sooner and performs comparably. The RL project is a research budget, not a delivery plan.