One operator rejects an agent after a backtest shows a 28% drawdown. Another approves a live deployment because a three-month paper run returned 11%.
Both calls can be wrong. The first figure may reflect a flawed historical simulation, while the second may be noise from a favourable market and an execution simulator that never had to fill a real order.
For software that places orders without a human watching, backtesting and forward testing answer different risk questions. Backtesting asks how a defined strategy would have behaved on historical data under stated assumptions. Forward testing asks how the running system behaves on newly arriving data, while it makes decisions and submits orders through a trading interface.
Neither test proves future performance or regulatory compliance. The useful task is to make each test falsifiable: define what can fail, collect evidence, and set conditions that stop the agent before a test failure becomes a live loss.
Separate strategy evidence from execution evidence
A backtest is good at checking repeatable questions about a strategy. Does the entry rule use information available at that time?
Does performance survive higher fees? Does the drawdown stay within a specified limit across different periods? These tests are cheap to rerun and make it practical to compare parameter sets, markets and risk assumptions.
But historical data usually records trades or quotes, not the exact conditions your order would have faced. A candle backtest may assume a fill at the close even though the signal only became available after the close.
A limit order may be counted as filled when the market touched its price, without modelling queue position, competing orders or partial fills. The code can produce a clean equity curve while hiding these assumptions.
Forward testing puts the strategy into a chronological stream it has not used for fitting. In paper trading, the system can send orders to a simulated venue. In shadow mode, it generates intended orders but does not submit them.
These approaches expose issues such as stale data, rejected orders, retries, clock drift and state recovery. They do not guarantee that live fills will match simulated fills.
Backtesting tests a model of the past. Forward testing tests a running system against the present.
Keep separate pass criteria. A strategy can pass a return or drawdown threshold in a backtest and still fail forward testing because the order manager duplicates a request after a timeout. Conversely, an orderly forward run cannot repair a strategy built on leakage or overfitting. Record both kinds of evidence rather than compressing them into one score.
Make backtest assumptions inspectable
Start by fixing the data and execution assumptions before trying parameter variations. Store the dataset version, timestamp convention, instrument mapping, fee schedule and code revision with every run. Use point-in-time data where possible.
For equities, that means accounting for delisted securities and corporate actions. For any market, check that the strategy cannot read a bar, quote or indicator before it would have been available to the process.
Run the signal against event-time data and make the simulated order eligible only after the signal could have been received and acted on. Include exchange calendars, trading halts, market hours, minimum order sizes, tick sizes and applicable funding or borrow costs.
If these inputs are missing, document the gap and test a conservative alternative. A result that disappears when fees rise modestly or execution moves by one bar is weak evidence for unattended deployment.
The figures below are typical ranges for liquid instruments in ordinary conditions, not audited results from a named provider. Actual values vary materially by venue, instrument, order size, volatility and connection route. Use them as prompts for stress tests, not as universal constants.
| Measure | Backtest assumption, typical range | Forward test observation, typical range | Risk question |
|---|---|---|---|
| Trading cost per side | 0.5 to 5 basis points, plus stated fees | 1 to 10 basis points, including spread and simulated slippage | Does the edge survive realistic costs? |
| Order response time | Often fixed or omitted, roughly 0 to 100 milliseconds | Roughly 50 to 500 milliseconds for a typical API round trip | Can a delayed response make the signal stale? |
| Limit-order fill rate | Often assumed at every eligible touch | Roughly 20% to 80%, depending on queue and order conditions | Does the test model partial or absent fills? |
| Rejected or timed-out requests | Often omitted or treated as successful | Roughly 0.1% to 2% during ordinary operation, with higher rates possible during incidents | Can the agent recover without duplicating exposure? |
Do not treat a single holdout period as protection against overfitting. If a researcher tests hundreds of parameter combinations, some will look strong by chance.
Keep a final untouched period, use rolling or walk-forward evaluation, and report the number of trials. Compare performance across regimes and instruments, not only across adjacent date ranges that share the same market conditions.
Use a distribution of outcomes rather than one best-case curve. Bootstrap or resample returns where appropriate, examine worst rolling drawdowns, and stress costs, missed fills and gaps.
These methods have limitations, particularly when returns are dependent or the market changes. They still make uncertainty more visible than a single annualised return.
Design forward tests to expose failure modes
A forward test needs a written protocol. Choose a duration or number of eligible signals before starting, define what counts as a valid order, and decide which incidents invalidate the run.
A few quiet days without signals do not provide much evidence. A test that covers a spread widening, a market opening, or an API interruption may reveal more than a longer run in uneventful conditions.
Use a fixed starting state and capture each decision from input to outcome. Store the market data timestamp, decision timestamp, intended order, submitted request, venue response, fill events, position after reconciliation and any risk-rule action. Retain rejected orders and cancelled orders too. If the event log cannot explain why the agent held a position, that is a control gap, not just a debugging inconvenience.
A staged protocol can reduce the chance that an early success hides an operational defect:
- Replay. Run deterministic historical scenarios, including gaps, duplicate messages, partial fills and rejected orders. Confirm that the same inputs produce the same decisions and state transitions.
- Shadow. Feed current data to the agent without submitting orders. Compare intended orders with the strategy's risk limits, and record stale data, missed signals and state mismatches.
- Paper. Submit orders to a simulated venue. Test order acknowledgements, cancellation, replacement, restart recovery and position reconciliation. Label all simulated fills as simulated.
- Limited live exposure. Only after explicit review, test with a small, pre-set exposure cap and real venue controls. Monitor the order and position streams, not only the strategy's target positions.
Paper trading can check request handling without exposing capital, but the simulator may not reproduce queue priority or market impact. A paper environment such as agentic paper trading can help exercise an agent against an order workflow before live use. Treat its output as test evidence with a scope, not as proof that live execution will be identical.
Set a stop condition in advance. Examples include a position mismatch that persists beyond a defined reconciliation interval, repeated timeouts, a daily loss limit, or missing market data beyond a chosen age.
A typical operational threshold might flag a quote older than 1 to 5 seconds for a short-horizon strategy, but the correct limit depends on the instrument and decision frequency. The agent should enter a defined safe state, such as cancelling open orders and blocking new risk, rather than improvising a recovery trade.
Keep risk and compliance evidence with the run
Compliance requirements depend on the firm, instrument, venue and jurisdiction. A test record does not itself establish compliance. It can, however, make review and incident reconstruction possible.
Preserve the strategy version, configuration, data source, clock source, risk settings, order events, approvals and test results. Use access controls and retention periods that match the organisation's policies and legal obligations.
Make controls independent of the strategy logic where practical. The strategy may suggest an order, but a separate order gateway can enforce maximum quantity, notional exposure, permitted instruments, price collars and order-rate limits.
A kill switch should block new orders reliably. Define who or what can activate it, what happens to existing orders and positions, and how the system is restarted after an incident.
For an unattended agent, an audit trail should allow an operator to reconstruct the chain from data to action. At minimum, capture:
- The input values and timestamps used for each decision, including feed delays or missing fields;
- The agent's signal, target position and risk checks, with the software and configuration versions;
- Each order request and response, including client identifiers, retries, amendments, fills and cancellations;
- Position and cash reconciliations, alerts, manual interventions and kill-switch events.
Protect the record from silent changes. Use append-only storage or equivalent access controls, synchronise clocks and alert when event sequences are incomplete.
Redact credentials and sensitive personal data. A log full of raw payloads can still be poor evidence if nobody can connect a response to the order that caused it.
Assign a person or team to review the test, even if the agent executes orders without a human in the loop. Define what an exception looks like, who receives an alert and how quickly the system must stop or escalate.
A paper run that silently continues after a position mismatch is evidence that a control failed. It should not pass because the simulated account ended in profit.
Set a decision rule before the agent trades
Use backtesting to reject fragile strategies and quantify uncertainty. Use forward testing to inspect timing, order-state handling, risk controls and operational resilience.
A typical programme may include several years of historical data where suitable, rolling holdouts, and weeks of forward testing, but neither a fixed number of years nor a fixed number of days is a universal standard. The required evidence depends on the strategy's signal frequency, turnover, market exposure and failure cost.
Write acceptance criteria in measurable terms. For example: no duplicate live order after a simulated timeout; all positions reconcile within a defined interval; the agent blocks new exposure when data is stale; and drawdown, turnover and cost sensitivity remain inside the risk budget.
State which exceptions require a restart or a new review. Avoid moving a threshold after seeing the results unless the change is documented and the affected test is rerun.
Forward evidence also decays. A change to the model, data feed, venue adapter, order logic or risk configuration can invalidate part of the earlier test.
Track changes and rerun the scenarios they affect. Keep the deployment limited to the instruments and order types that were actually tested; a market order path does not validate a new use of passive limits.
This week, write and approve a one-page test protocol for one agent, listing its data, pass thresholds, stop conditions and required event logs.
Frequently Asked Questions
How long should a trading agent forward test run?
Set the duration or number of eligible signals before testing begins. A few weeks may expose operational faults, but a low-frequency strategy may need more time to produce useful evidence. Include varied market conditions and relevant failure scenarios rather than relying on elapsed time alone.
Can paper trading replace backtesting?
No. Paper trading tests a running system and its order workflow, while backtesting evaluates a strategy against historical data and assumptions. Use both, and record which execution behaviours the simulator cannot reproduce.
What should stop an unattended trading agent?
Block new risk when a pre-defined critical condition occurs, such as stale market data, an unreconciled position, repeated order timeouts or a breached loss limit. Specify whether the agent should cancel open orders, alert an operator or enter another safe state, then test that response before deployment.
