If the strategy only works when every order fills at the price in the bar, the next test is not another parameter search. It is a forward run that exposes the gap between a signal and an actual order lifecycle. Backtesting estimates whether an idea might have an edge. Forward testing asks whether the deployed agent can observe, decide, submit, reconcile and stop safely under current conditions.
These tests answer different questions. Treating a paper account as proof of profitability, or a historical simulation as proof of operational reliability, confuses them. A useful 30-day plan gives each test a separate job and a stop condition.
Set a boundary between the tests
Backtesting replays historical market data through strategy and execution assumptions. It is fast, repeatable and useful for rejecting weak ideas.
Forward testing runs the frozen agent against new market events. It is slower, but it can reveal stale data, API timeouts, unexpected order states and behaviour that never appeared in the historical sample.
Neither is a substitute for the other. A backtest can model queue position, but it cannot know the queue your future order would have joined.
A paper venue can return fills, but its fill model may not reproduce a live venue's priority, market impact or rejection rules. The practical aim is to find disagreements between the two, then identify their cause.
A strategy is not validated until its decisions and its order handling have both survived a test they could fail.
Separate the data used to tune the strategy from data used to evaluate it. Use chronological splits, not a random shuffle: for example, train on 18 months, validate on six, then reserve the latest six months as an untouched test.
Include delisted instruments where relevant, corporate actions, trading calendars and point-in-time data. A current constituents list applied to old prices creates survivorship bias.
Log the agent's inputs and outputs on every decision. Record data timestamps, feature values, model or prompt version, intended quantity, risk checks and resulting order events. A decision that cannot be replayed from these records cannot be debugged reliably.
Make the backtest pay for execution
Build the order model before optimising the signal. Simulate the same state machine that will run in production: submit, acknowledge, partially fill, fill, cancel, reject and reconcile.
A timed-out request is not automatically a failed order. The venue may have accepted it, so the agent must check order status before retrying. Model fees, spread, slippage, minimum order size, tick size, trading hours and rate limits.
Use the highest-quality data the strategy's horizon requires. Daily bars are inadequate for an agent reacting to intraday quotes. Bar-based tests also need explicit assumptions about whether a stop or limit could have filled when a bar crosses its price. If the sequence of high and low within a bar is unknown, choose a conservative fill rule or use finer data.
Run sensitivity tests instead of trusting one cost setting. If a strategy only survives with zero latency, perfect fills or costs below plausible market levels, it is not ready for forward evaluation. Typical ranges below are illustrative, not guarantees. They vary with instrument, venue, order size, time of day and market conditions.
| Measure | Typical backtest treatment | Typical forward observation |
|---|---|---|
| Spread cost for liquid US shares | 0-2 basis points round trip in a simple model | 1-10 basis points round trip when crossing quotes |
| Market impact for small orders | 0-5 basis points per side, often assumed | 1-20 basis points per side, with higher costs in thin periods |
| Order acknowledgement latency | Zero or a fixed delay | 50-500 ms is a typical network range; spikes can exceed one second |
| Passive limit fill on a price touch | 80-100% in permissive touch-fill models | 0-60% of touch opportunities, depending on queue and venue |
These figures are prompts for stress tests, not universal execution benchmarks. Measure the distribution in the forward run and compare like with like: same instrument, session, order type and participation rate.
Track median and 95th percentile latency, not only the average. A small number of slow responses can matter more than a tidy mean.
Use forward tests to inspect the whole agent
Forward testing should progress through distinct levels. Start with shadow mode, where the agent receives live data and records intended orders but sends none. Next, use paper trading to exercise API calls, order states and recovery logic. Only after those gates pass should a controlled live test be considered, with a fixed small risk budget and explicit human escalation rules.
Paper tests need their own limits. They can verify that the agent handles a partial fill, cancels a stale order and restores state after a restart. They cannot establish that a live market order will get the same price. If a paper venue is part of the test environment, check what its simulator represents and what it omits. Agentic Trading Index offers agentic paper trading from $9.99 a month for developers who need a place to run these workflows.
Keep a daily comparison between the backtest's expected behaviour and forward observations. Useful fields include signal time, decision time, submit time, acknowledgement time, fill time, requested quantity, filled quantity, price, fees and reason for rejection. Also record missing or late market data and every restart. A divergence is a diagnostic, not an invitation to tune against the forward sample.
Set gates before the run. Example thresholds for a small liquid-market agent might include: no duplicate live orders after a timeout or restart; 100% of submitted orders reconciled to a terminal or explicitly open state; no risk-limit breach; and a 95th percentile decision-to-acknowledgement latency below a chosen limit such as 500 ms.
Set slippage and drawdown limits from the strategy and venue, not from a generic template. If a gate fails, pause and investigate rather than silently relaxing it.
Run the 30-day implementation plan
Freeze the test specification before day one. Use the same code path for simulation and forward runs where practical, but make the execution adapter explicit so simulated fills cannot be mistaken for venue events. Version the agent, configuration, data snapshot and risk limits for every run.
- Days 1-5: define and instrument. Write down the hypothesis, instruments, decision horizon, allowed order types and risk budget. Add structured logs for inputs, decisions and order events. Define pass and fail thresholds for duplicate orders, stale data, latency, reconciliation and drawdown.
- Days 6-10: establish a clean backtest. Check timestamps, calendars, corporate actions and point-in-time universe membership. Split data chronologically. Include fees and conservative fill rules, then run at least three cost scenarios. Save the exact code and configuration used for the untouched test period.
- Days 11-15: test failure paths. Exercise rejected orders, partial fills, delayed acknowledgements, rate limits, dropped connections and process restarts. Inject a timeout after submission and confirm the agent queries order status before retrying. Test that stale prices block new orders and that kill conditions cannot be bypassed by a restart.
- Days 16-23: run shadow and paper tests. Keep the strategy frozen. Compare intended actions with simulated fills each day. Review mismatches, state transitions and tail latency. Fix software defects, but log each code change and restart the relevant test clock for affected gates.
- Days 24-30: review and decide. Reconcile every order event and calculate slippage, fill rate, fees, rejected orders and risk-limit incidents. Compare results with the pre-set thresholds. If all operational gates pass, decide whether a separately capped live pilot is justified. A short paper run is not evidence of a durable trading edge.
At day 30, the useful output is not a single return number. It is a versioned record of what was tested, what failed, what changed and which risks remain unmeasured.
If the strategy's edge is small relative to execution costs, more testing may show that the idea should be discarded. If the agent fails operational gates, keep it out of live trading even if the backtest looks strong.
Decide what the evidence permits
Compare results with the original hypothesis, not with a moving target. Forward observations are a new sample. Do not use them repeatedly to select parameters and then report them as independent validation. If a change is needed, mark the old test as consumed, make the change explicit and collect a fresh forward sample.
Separate strategy performance from system reliability. A profitable paper run can conceal optimistic fills. A losing forward run may reveal a sound signal but poor execution, or it may simply be ordinary variance.
Check order-level costs and state transitions before changing the model. For low-frequency strategies, 30 days may not contain enough trades to assess returns. It can still test deployment mechanics, but not establish statistical confidence in the edge.
Only move to live orders when the agent can recover its position and open-order state from authoritative records, enforce a hard exposure limit, and stop sending orders when data or reconciliation is uncertain. Keep the live pilot's notional and loss cap low enough that a failed assumption is survivable. Define who or what receives an alert, and what action follows if the agent cannot resolve an order.
Stop doing these things:
- Stop treating a price touch as proof of a limit fill.
- Stop changing parameters during an evaluation run.
- Stop retrying timed-out orders without checking their status.
- Stop calling paper returns proof of live profitability.
Frequently Asked Questions
What is the difference between backtesting and forward testing?
Backtesting replays historical data using a strategy and execution model. Forward testing runs the agent on new market data, exposing current data, API and order-handling behaviour that historical replay cannot reproduce.
How long should a trading agent be forward tested?
Use enough time to exercise the relevant market conditions and failure paths. A 30-day run can test deployment mechanics, but low-frequency strategies may need substantially longer to collect enough trades to assess performance.
Can paper trading prove a strategy is profitable?
No. Paper trading can test decision flow, order-state handling and recovery logic, but simulated fills may not reproduce queue priority, market impact or live slippage. Treat paper returns as a limited diagnostic.
What should a backtest include for an automated trading agent?
Model fees, spread, slippage, order types, partial fills and realistic order-state transitions. Use point-in-time data and chronological splits, and test costs under more than one plausible assumption.
When should an agent move from paper to live trading?
Move only after it passes pre-set operational gates, reconciles all orders and positions, and enforces exposure and loss limits. Start with a capped live pilot; a successful paper run alone does not establish a trading edge.
