At 17:05 on Friday, a developer is staring at a dashboard after the agent has lost money for five sessions. The backtest shows a smooth equity curve. The live log shows missed entries, two rejected orders and a position that stayed open after a reconnect. The question is not whether the strategy was profitable in the historical run. It is which part of the system failed to behave as tested.
Backtesting and forward testing answer different questions. A backtest asks how a specified policy might have behaved on past data under a model of execution.
A forward test asks how the running system behaves on new data as events arrive. Neither test, on its own, proves that an unattended agent is safe or profitable. The common mistake is treating the second as a routine confirmation of the first, rather than an experiment that can expose a different class of failure.
Do not read a loss as a strategy verdict
A weekly loss is an outcome, not an explanation. Split the result into signal quality, sizing, execution and operations before changing parameters. Compare intended orders with broker acknowledgements, fills, fees and final positions. If the agent wanted to buy 100 shares, received a partial fill for 40 and then cancelled the remainder, the signal did not necessarily fail.
The execution policy may have. If the broker filled the order but the agent's local position still says zero, this is an accounting or reconciliation fault.
Use a per-decision record that can reconstruct what the agent knew and did. Store the market-data timestamp and receipt time, feature values, model and policy versions, target position, risk checks, order request, broker response, fills, cancellations and resulting position.
Use a stable decision ID and client order ID. Without that trail, a chart of daily returns cannot distinguish a bad forecast from stale quotes, duplicate submissions or a retry sent after a timeout.
Compare outcomes at the same level. A backtest using mid-price returns is not comparable with a live result after spread, fees and market impact.
Nor is a bar-close strategy comparable with a live system that sees the bar only after its close and then waits for a network round trip. Fix the decision timestamp, price convention, sizing rule and cost model before judging performance. Otherwise, a favourable difference may be a measurement mismatch, not evidence that one test is better.
Backtests reward shortcuts unless the harness resists them
Historical data is not a neutral replay of the market. Bar data can hide the order of events inside a candle.
A stop and a profit target may both appear to have traded, even though the bar does not reveal which came first. A strategy that uses the final close to decide and also assumes a fill at that close has look-ahead bias. For intraday systems, use event or tick data where the decision depends on intrabar order, or apply conservative assumptions when the sequence is unknown.
Check the data pipeline for survivorship bias, revised corporate actions, missing delisted instruments, timezone errors and duplicate events. Features must be calculated only from information available at the decision time.
A rolling average, normalisation window or universe filter can leak future information if it is fitted across the whole dataset before splitting. Keep a final holdout period untouched until the strategy and its parameters are frozen. Repeatedly checking that holdout turns it into another training set.
Agent strategies add a second source of overfitting: the decision system may change even when the trading idea does not. Record the model identifier, prompt or policy text, tool definitions, sampling settings, code commit, dependency versions and random seed where applicable. Cache model outputs during a comparison. If the same backtest changes on rerun, first establish whether the cause is stochastic inference, data ordering or a software change. Do not attribute the difference to market conditions.
Execution assumptions deserve their own stress tests. A market order at the next bar's open is not a realistic fill model for every asset or order size. Limit orders need queue assumptions, partial fills and a rule for cancellation. Model fees, spread and slippage separately, then test a wider range than the best estimate. Typical market ranges vary by venue, instrument and liquidity, but a useful first pass is to stress slippage at 1.5 to 3 times the base estimate and latency at 2 to 5 times the measured median.
- Run a leakage check: shift features forward and confirm performance collapses rather than improves.
- Compare results across time periods, instruments and parameter neighbourhoods, not just the best configuration.
- Report turnover, maximum drawdown, exposure, rejected-order assumptions and costs beside return.
- Keep an audit file of every strategy version tested. Hundreds of trials make a single attractive Sharpe ratio weak evidence.
A useful backtest is a filter, not a green light. Reject a candidate if small changes to fees, start dates or parameters destroy its performance. The test should make it easy to find where the idea breaks, rather than easy to produce a compelling equity curve.
Forward tests check the running system, not just the signal
Forward testing means running the frozen strategy on data that arrives after the test begins. In shadow mode, the agent computes decisions but sends no orders. In paper trading, an execution simulator records hypothetical orders and fills.
A small live canary sends real orders under strict limits. These stages are not interchangeable. Shadow mode tests decision timing and operational stability, paper mode adds an execution approximation, and live orders reveal actual venue responses, fees and market impact.
The figures below are typical market ranges for planning a test, not guarantees or audited results. Instrument liquidity, trading frequency and venue behaviour can move them substantially. A low-frequency strategy may need months to collect enough independent decisions, while a high-frequency agent can produce a misleadingly large sample in a short period if those observations are strongly correlated.
| Test layer | Typical range | What it can expose |
|---|---|---|
| Historical backtest | 3 to 10 years; 1-minute bars to tick data | Regimes, parameter sensitivity and broad cost assumptions |
| Event replay | 1 to 20 sessions; 10 to 250 ms simulated decision delay | Event ordering, clock handling and order-state logic |
| Paper forward test | 4 to 12 weeks; 100 to 500 or more eligible signals | Live data handling, scheduled jobs and simulated order lifecycle |
| Small live canary | 1 to 4 weeks; 1% to 5% of planned order size | Actual fills, rejects, fees, impact and broker-specific behaviour |
Do not confuse the number of signals with statistical certainty. One hundred trades can still leave a wide range of plausible win rates, and consecutive trades in the same market regime are not independent observations.
Track the conditions around each decision: spread, volatility, liquidity, market hours and data quality. A test that spans different conditions is more informative than a larger count from a single quiet stretch.
Paper fills are estimates. A simulator may fill a limit order as soon as the market touches its price, while a live order would sit behind the queue or never fill. It may not model exchange throttles, price bands, auction behaviour or a broker's cancel-replace rules. Paper performance is useful for testing order construction and state transitions, but should not be presented as a forecast of live returns. A paper environment such as the Agentic Trading Index paper-trading environment, available from $9.99 a month, can help exercise those workflows before risking capital.
Build a sequence with explicit failure limits
Move a candidate through gates. Each gate should have a written pass condition, a failure condition and a named version of the software under test.
Keep the strategy frozen between stages. If a fix changes the decision logic, risk sizing or order handling, rerun the affected tests instead of treating the new build as the old candidate.
- Define the claim. Write down the instrument universe, decision frequency, data needed, target holding period and intended order types. Specify what success means in net terms, including fees and slippage. Record operational limits such as maximum position, daily loss and order rate.
- Freeze the candidate. Pin the code commit, model version, prompts, configuration, dependencies and data snapshot. Set deterministic seeds where possible. Save a machine-readable manifest with every run so a result can be reproduced.
- Validate the historical harness. Test timestamps, corporate actions, missing data, feature availability and fill rules. Use chronological splits, keep one holdout untouched, and run adverse cost and latency scenarios. Reject candidates that rely on a narrow parameter peak.
- Replay event sequences. Feed market events and broker-like responses in timestamp order. Inject partial fills, rejects, delayed acknowledgements, duplicate messages, disconnects and restarts. Assert that retries are idempotent and that the agent reconciles broker positions before sending new risk.
- Run shadow and paper forward tests. Compare each intended action with the data available at that moment. Check stale-data detection, clock drift, rate limits, order-state transitions and recovery after process restarts. Set a minimum elapsed period and a minimum number of relevant market conditions, not just a target number of profitable trades.
- Use a bounded live canary. Start with a small fraction of planned size and hard caps on position, notional, order frequency and daily loss. Monitor actual slippage against a pre-set tolerance. Stop automatically on position mismatch, repeated rejects, stale market data, loss-limit breach or loss of the broker connection.
- Review before scaling. Reconcile every order and fill, compare live costs with the test assumptions, and document unexplained differences. Increase size in steps only after the system meets its pre-written criteria. Any change to model, execution logic or risk controls starts a new validation record.
For unattended software, the stop conditions matter as much as the performance threshold. A process should fail closed when it cannot establish its current position, market-data freshness or order state.
Alerts are useful, but an alert that depends on a human replying is not a control. Define what the agent does on its own: stop opening risk, cancel working orders where safe, reconcile state and wait for a defined recovery condition.
Forward testing is not a ceremony after backtesting. It is the point where assumptions meet event timing, broker behaviour and operational faults. The mistake most readers will still make is treating a longer forward test as proof, instead of asking whether it actually exercised the failure modes that could stop the agent trading safely.
Frequently Asked Questions
How long should I forward test a trading agent?
A typical paper forward test may run for 4 to 12 weeks, but elapsed time alone is a poor measure. Set a minimum number of relevant signals and market conditions, then extend the test if the strategy trades infrequently or has not encountered stressed conditions.
Is paper trading enough before an agent goes live?
No. Paper trading can test data handling, order construction and state transitions, but simulated fills may not reflect queues, rejects, fees or market impact. A tightly capped live canary is needed to observe actual venue behaviour.
What is the main difference between backtesting and forward testing?
Backtesting applies a strategy to historical data using assumptions about fills and costs. Forward testing runs the strategy on newly arriving data, which can reveal timing, connectivity, order-state and execution problems that historical simulation misses.
How do I prevent data leakage in a trading backtest?
Calculate each feature only from information available at its decision timestamp, and split data chronologically. Keep a final holdout untouched until the candidate is frozen, and check for revised data, survivorship bias, timezone mistakes and preprocessing fitted across future observations.
What should stop an unattended trading agent automatically?
Use hard triggers for stale market data, position mismatches, repeated order rejects, connection loss, excessive order rates and daily loss limits. The agent should stop opening new risk and follow a defined recovery procedure without requiring a human to notice the alert first.
