Backtesting versus forward testing for agent strategies

Two operators, opposite calls

One operator sees a backtest with a 1.8 Sharpe ratio and authorises an unattended strategy for live orders. Another sees the same curve, but notices it assumes fills at the mid-price and uses end-of-day signals that arrive before the recorded close.

That operator rejects the launch and asks for forward testing first. The disagreement is about evidence, not appetite for risk.

Backtesting replays a strategy against historical data. It is useful for checking whether a rule could have worked across different market conditions, and for finding bugs before they touch an account.

Forward testing runs the strategy against new market data as it arrives. In paper mode, orders are simulated; in live mode, they reach a venue or broker and can lose money. Neither test can stand in for the other. Historical replay gives a larger sample quickly, but depends on data quality and execution assumptions.

Forward testing measures the running system, including feed delays, state management and order responses, but may produce too few trades to establish statistical confidence. An unattended agent needs both, with separate pass criteria.

Start by defining what the strategy can observe at each decision point. If it uses a bar close, the signal cannot act on that close unless the data and execution timestamps make that possible.

A backtest that fills at the same close after calculating from the final close price has look-ahead bias. A forward test can reveal timing defects, but only if logs preserve event time, receive time, decision time and order acknowledgement time.

What each test costs, and what it measures

Testing costs include data, compute, engineering time and, for live validation, capital exposed to market risk. The figures below are typical market ranges for a modest strategy, not audited results for any company. Data fees depend on asset class, exchange rights and whether the feed is delayed, consolidated or tick-level. A developer's own labour is often the largest cost.

These costs do not include engineering time. A common contractor billing range is $50-$150 an hour. Data cleaning, replay tooling and order-state instrumentation can take tens of hours before a strategy is ready for a useful test. That effort is often more valuable than buying a finer-grained dataset before the system can consume it correctly.

Paper environments also vary in cost and realism. Agentic Trading Index offers agentic paper trading from $9.99 a month through its paper trading service. Check whether a test environment models partial fills, rejections, rate limits and market hours. A low subscription price does not make its simulated fills equivalent to live execution.

For liquid US equities, a rough research assumption might put spread and slippage at 1-10 basis points per round trip in normal conditions. Less liquid shares, fast markets and market orders can cost substantially more. These are typical ranges, not promises. Measure costs by instrument, order type and time of day, and include commissions, exchange fees, borrow charges or funding where relevant.

Build a benchmark that can fail

A useful comparison is not "did the paper account make money?" It is whether the observed behaviour stays within declared tolerances. A profitable week can conceal stale prices, a missed stop or a fill model that grants the agent prices it could not trade. Compare the same signals and intended orders across replay and forward runs, then account for differences in available data and execution.

1. Freeze the test inputs. Record strategy version, configuration, data vendor, universe and timestamps. Keep a holdout period that was not used to tune thresholds. For a portfolio strategy, preserve the historical constituents rather than applying today's survivors to the past.
2. Replay with realistic timing. Apply latency between observation, decision and submission. Use bid and ask prices for marketable orders, model queue uncertainty for passive limits, and include fees. If the historical data cannot support a fill assumption, mark the fill as uncertain instead of awarding it automatically.
3. Run forward in paper mode. Let the same production code process live or delayed feeds and emit the same order intents it would send to a broker. Log intended price, submitted price, acknowledgement, fills, cancels and final position. Test reconnects and duplicate messages, not only normal sessions.
4. Compare enough observations. Use a minimum test window suited to the strategy's frequency. As a starting point, 20-60 sessions and 100-300 order events can expose operational defects, but may not establish that returns are repeatable. A weekly strategy may need months to produce a meaningful sample.
5. Set gates before the run. For example, require median realised slippage to remain below 1.5 times the backtest assumption, with the 95th percentile below twice that assumption. Set a separate threshold for rejects, duplicate orders and unhandled position mismatches. Choose limits from the strategy's expected edge and risk budget, not from a generic pass mark.
6. Investigate every breach. Separate market movement from software errors. Re-run the affected period with recorded inputs, and do not retune on the forward sample then count it as independent validation.

Track operational measures alongside returns:

• Signal-to-order latency, including median and 95th percentile.
• Fill ratio and partial-fill rate by order type and venue.
• Rejects, cancels, timeouts and duplicate submissions per 100 orders.
• Difference between intended and realised position after every event.
• Realised fees and slippage compared with the assumptions used in replay.

Decide what the evidence permits

Backtesting is best for breadth. It can cover years of regimes, stress parameter sensitivity and reveal whether a strategy's edge depends on a narrow date range. Use walk-forward splits: fit or select parameters on one period, then evaluate on the next untouched period. Repeat the process through time. A single train-test split is easy to overfit, especially when many signal variations have been tried.

Forward testing is best for the live pipeline. It checks whether the agent receives the expected fields, handles asynchronous responses and maintains a correct position after partial fills or reconnects.

It also shows whether paper execution is optimistic. It does not prove profitability: a few dozen trades have wide uncertainty, and simulated fills can still differ from venue behaviour.

Before enabling small live orders, compare forward results with the backtest by trade, not just by aggregate return. Inspect trades whose entry or exit differs materially.

Separate strategy drift from execution drift: a changed signal count may indicate feed or logic problems, while worse fills with similar signals point to cost assumptions or order design. If the strategy's expected edge is 8 basis points per trade and realised costs are 10 basis points, more paper sessions will not fix the economics.

Use a staged release. First permit signals without orders. Then run paper orders with production code and alerts. Next, allow a limited live notional, restrictive order limits and a tested kill switch.

Increase size only after fills, positions and risk checks reconcile consistently. Keep a human escalation path for conditions the agent cannot classify, such as repeated broker timeouts or a position that differs from the internal ledger.

For each stage, write down the evidence required to advance and the conditions that stop the run. Keep the test logs and exact software version so a later change can be compared fairly.

A strategy is not validated merely because its backtest passed or its paper balance rose. It is ready for the next stage only when its expected costs, observed behaviour and failure handling fit the risk budget. Check your last 100 order events for the actual reject rate.

#paper trading #forward testing trading strategy #trading execution costs #trading agent validation #automated trading backtest #walk-forward analysis #agent strategy testing #paper trading benchmarks