Devin Hartcastle
At the end of a bad week, a developer looks at a dashboard showing a 4% drawdown, several rejected orders and fills that differ from the strategy's expected prices. The backtest had a smooth equity curve. The live-looking system now has a question the chart cannot answer: is the strategy wrong, or did the test miss how orders behave?
Backtesting and forward testing answer different questions. Backtesting replays rules against historical data.
Forward testing runs the same code on incoming market data, usually with simulated orders or a broker's paper account. An agent that places orders without a human watching needs both, plus a record of every decision and order event.
What each test can tell you
A backtest is useful for checking whether a rule had a plausible edge across different market conditions. It can help find coding errors, compare parameter sets and estimate turnover, drawdown and exposure. It cannot recreate a live order queue or prove that a broker API will respond as expected.
Forward testing runs the deployed strategy against current data. It reveals problems that historical replay can conceal: a stale quote, a clock mismatch, an order rejected for insufficient buying power, or a retry that submits the same intent twice. It also tests whether the agent's logs let an operator reconstruct what happened.
Neither test proves future profitability. Historical results depend on data quality and assumptions; forward results depend on the short period observed and the fidelity of the paper environment. Treat them as separate evidence, not competing versions of the same score.
- Backtest: repeatable and fast, but only as realistic as its data, fill model and costs.
- Forward test: closer to live operations, but often too short to establish an edge and potentially unlike live execution.
- Live trading: the only place to observe real fills and capital risk. Keep initial exposure small and controlled.
Model the parts a chart leaves out
Bar data may show that a price touched a limit order. It does not establish that the order would have filled. A bar-based test that fills every touched limit can overstate performance, especially when the strategy depends on queue position or trades frequently. Use tick or quote data where available, and make fill assumptions explicit.
Include commissions, exchange and regulatory fees where relevant, bid-ask spread, slippage and financing costs. Check corporate actions, delisted instruments, futures rolls and the timestamp convention for each data source. A signal calculated using a bar's closing price must not also trade at that same close unless the data and order timing support it.
The table gives typical market ranges for liquid instruments and common API workflows. They are orientation points, not guaranteed costs. Asset, order size, venue, volatility and broker design can move actual values well outside them.
| Measure | Backtest assumptions, typical ranges | Forward-test observations, typical ranges | What to inspect |
|---|---|---|---|
| API round-trip latency | 0-100 ms in a local simulator | 50-500 ms in ordinary conditions; 1-5 seconds or more during delays | Measure from request send to acknowledgement, not just strategy runtime. |
| Spread on liquid shares | 0-5 basis points, often fixed or inferred | 1-5 basis points in ordinary conditions; 5-20 or more in volatile periods | Compare the decision quote with the executable bid or ask. |
| Market-order slippage | 0-5 basis points if modelled; sometimes omitted | 0-5 basis points in ordinary conditions; 5-30 or more during fast markets | Group by order size, volatility and time of day. |
| Limit-order fill rate | Can be 100% whenever a bar touches the limit | Highly variable; a paper simulator may fill 70-100% of eligible orders | Check whether the simulator models queue position and partial fills. |
These ranges are not universal, and paper fills can be more favourable than live fills. For a separate agentic paper-trading environment, check what order events and execution assumptions it exposes before relying on its results.
Run a forward test that can fail usefully
Freeze the strategy version before starting. Record a version identifier, configuration, data timestamps and market calendar. The forward run should use the same signal code, sizing logic, risk checks and order-state handling intended for deployment. If you change parameters mid-run, mark a new version rather than blending the evidence.
Give the agent a paper account or a no-send mode that exercises the complete order path. Capture each signal, intended order, request, acknowledgement, fill, cancellation and rejection with a common timestamp and unique intent identifier. Reconcile the agent's position against the broker or simulator after every order event and on a regular schedule.
- Set the question. For example, test whether the strategy's net performance survives realistic costs, or whether the order manager handles partial fills correctly.
- Define limits before running. Set maximum position, order notional, daily loss and permitted instruments. Specify what trips a halt and what state the system must enter.
- Run the frozen build. Keep a record of market-data gaps, reconnects, rejects and manual interventions. Do not silently repair bad runs.
- Compare expected and observed execution. Calculate spread paid, slippage, acknowledgement delay, fill rate and reject rate by instrument and order type.
- Review failures, then rerun. Fix defects, increment the version and repeat the test. Do not treat a repaired run as evidence for the earlier build.
Choose a duration and trade count that expose different conditions. A few days may catch a broken endpoint but say little about a strategy's edge. Several weeks and at least 100 signals can reveal recurring execution and operational issues, but correlated trades still provide less statistical evidence than 100 independent observations.
Set gates before live access
Agree on what acceptable means before looking at results. An operator might require no duplicate orders, complete event logs, successful position reconciliation, and tested halt behaviour.
For execution, compare the forward-test costs with the backtest's assumptions. If a strategy expects 2 basis points of gross edge per trade but observed costs commonly approach that amount, the historical result is not a sound basis for increasing exposure.
Separate software readiness from strategy performance. A strategy can have poor returns while the order system behaves correctly; a profitable paper period can still hide broken recovery logic.
Test outages, stale data, rate limits, partial fills and rejected orders deliberately. Confirm that a restart does not forget open orders or submit an old intent again.
Paper trading may not reproduce queue priority, market impact, borrow availability or live buying-power checks. Log those gaps and account for them in the live plan. If live access is eventually granted, begin with tightly capped size, monitor execution against predefined limits and retain a kill switch that stops new orders without losing track of existing positions.
This week, run one frozen strategy build in forward test and save its complete signal-to-fill event log.
Frequently Asked Questions
What is the difference between backtesting and forward testing?
Backtesting replays a strategy against historical data and simulated execution assumptions. Forward testing runs the strategy on incoming market data, usually with paper orders, so it can expose current API and order-management behaviour.
How long should I forward test a trading agent?
Run it long enough to observe varied market conditions and collect a useful number of signals. Several weeks and at least 100 signals can reveal recurring defects, but do not by themselves establish that a strategy has a durable edge.
Can paper trading prove that a strategy will be profitable live?
No. Paper execution may not reproduce queue position, market impact, borrowing constraints or live buying-power checks. Use paper results to validate code and estimate execution, then treat live performance as a separate test.
What should I measure during forward testing?
Track slippage, spread paid, acknowledgement latency, fill and reject rates, data gaps, duplicate intents and position reconciliation. Store event-level logs so a trade can be traced from signal through order state.
When is a trading agent ready for live orders?
Set explicit operational and risk gates before the test begins. Require reliable reconciliation, tested halt and recovery behaviour, and execution costs within the strategy's assumptions; start live access with capped exposure.
