A broker's paper account is not realistic simply because it uses the broker's live prices.
Market data is only one input to execution. An unattended agent also needs credible order acknowledgements, partial fills, rejects, cancel behaviour, fees, buying-power checks and position updates. Different test methods reproduce different parts of that chain. Treating them as interchangeable leaves gaps that may only appear after an agent starts placing live orders.
Know what the test must reproduce
Start with the failure modes that could change a position or leave the system uncertain. A limit order can rest unfilled while the price touches its limit. A market order can execute across several price levels. A cancel request can arrive after a fill, and a network timeout can leave the client unsure whether an order was accepted.
A credible test should model the full order lifecycle, not just a submitted order and a returned price. Record the request, acknowledgement, exchange or broker order identifier, every fill, fees, cancellation result and final position. For each event, retain timestamps and the source of the event. This makes it possible to tell whether an apparent strategy loss came from the signal, a fill assumption or a state-management defect.
Set acceptance criteria before comparing tools. For example, require that repeated requests cannot create duplicate exposure, that a rejected order does not alter the position, and that position totals reconcile after partial fills and cancels. Those checks apply whether the simulated trade makes money or not.
Broker sandboxes test the connection
A broker sandbox is usually the quickest way to test authentication, request formats, order types and basic account responses. It is valuable for integration work, particularly for checking how the API represents statuses such as accepted, partially filled, cancelled and rejected. Some sandboxes also expose rate limits and maintenance errors that a local mock would miss.
Its weakness is the simulated market. A sandbox may fill a limit order as soon as the displayed price touches the limit, with no queue ahead of it.
It may accept order sizes that would move a thin book, or omit borrow availability, exchange fees and auction behaviour. A typical API response time can range from tens to hundreds of milliseconds, but a quick sandbox response does not establish live execution quality.
Use the sandbox to test:
- Authentication renewal, request validation and rate-limit handling.
- Order-state transitions, including rejects and partial fills.
- Timeout recovery, duplicate-request protection and position queries.
Do not use its fill prices as a forecast of live slippage. Where the sandbox has simplified fills, document the simplification and cover it with a separate test.
Local simulators expose their assumptions
A local simulator gives the team control over matching rules. It can model market and limit orders, partial fills, fees, spread, slippage, latency, trading halts and position limits. It can also replay the same price and order event sequence repeatedly, which is useful for debugging an agent that behaves differently after a timeout or restart.
That control comes with responsibility. A simulator that fills every touched limit order is optimistic. A simple penalty such as adding one tick of slippage may be too harsh for a liquid future and too generous for a thin equity. Queue position is difficult to infer from ordinary bar data. With only one-minute candles, the simulator cannot reliably determine which order arrived first or how much volume traded at the limit.
Use the best data available for the question being tested. Quote or trade data helps with intrabar sequencing, while depth data can support rough queue estimates. Even then, queue estimates are approximations unless the feed and venue expose sufficient order-level information. For every model, keep the assumptions in configuration and run sensitivity tests across plausible ranges, such as 0 to 2 ticks of extra slippage or 50 to 500 milliseconds of simulated delay.
Shadow mode sees prices, not your fills
In shadow mode, the live strategy generates orders but does not send them to the market. The system records what it would have submitted and compares that decision with live prices and subsequent trades. This is useful for checking signal timing, market-data handling and operational stability under current conditions.
Shadow mode cannot reveal whether the hypothetical order would have filled. A limit order that appears marketable in a quote snapshot may lose its opportunity before arrival.
A large order may consume several levels, and a live order may be rejected because of buying power, price bands or a venue rule. Without actual execution, these remain modelled outcomes, not observations.
Keep the comparison explicit:
- Log intended submission time, order type, limit price and size.
- Calculate a range of plausible fills, not a single assumed fill.
- Separate signal performance from estimated spread, fees and slippage.
Shadow mode complements a simulator and a broker sandbox. It is not a substitute for testing order acknowledgements or cancellations.
Hosted environments trade setup for control
A hosted paper-trading service can reduce the work required to connect an agent, store runs and inspect order events. The trade-off is that the team has less control over matching logic and may have to adapt its application to the service's supported assets, order types and event format. Check whether the service exposes raw events and lets you export enough data to reproduce a discrepancy.
Some hosted tools provide a practical starting point without requiring an internal matching engine. For example, agentic paper trading is available from $9.99 a month. Check the current plan details and test the specific order behaviour your agent depends on before treating any service as venue-equivalent.
The figures below are typical market ranges for planning, not quotes or audited costs from a named provider. They exclude staff time after initial setup and can vary with data access, asset class and test volume.
| Approach | Typical setup effort | Typical monthly cash cost | Execution fidelity |
|---|---|---|---|
| Broker sandbox | 0.5 to 3 engineer days | $0 to $200 | Low to medium, API behaviour varies |
| Local simulator | 3 to 15 engineer days | $0 to $300, excluding paid data | Configurable, limited by inputs and model |
| Shadow mode | 1 to 5 engineer days | $0 to $500 or more for data and infrastructure | Live market context, no observed fills |
| Hosted paper service | 0.5 to 3 engineer days | $10 to $200 | Depends on supported rules and event detail |
Choose by failure mode, then verify
Most teams need more than one approach. A broker sandbox checks API integration. A local simulator makes fill assumptions testable. Shadow mode checks the strategy against live conditions. A hosted service can reduce infrastructure work. Choose a primary environment based on the risk that would be most costly in your system, then use the others to cover what it cannot observe.
Before allowing unattended live orders, run this checklist against a fixed test set:
- Submit market, limit, stop and cancel requests, including invalid prices and sizes.
- Simulate partial fills followed by more fills, a cancel or a reject.
- Drop the response after submission, then verify that retry logic cannot create duplicate orders.
- Restart the process and reconcile open orders, executions, cash and positions from the venue or broker.
- Compare estimated fills under several spread, latency, slippage and queue assumptions.
- Confirm that fees, minimum order sizes, buying power and market hours match the target account rules.
Keep a test report with the input data, simulator version, assumptions and event trace. A changed matching rule or API version should trigger a rerun, not a silent update to the agent's expected behaviour.
If this were my operation, I would first replay a recorded market session through a local simulator and compare its order-state trace with the broker sandbox. I would hold back live orders until duplicate protection, partial fills, cancels and position reconciliation all pass the same repeatable tests.
Frequently Asked Questions
What makes paper trading realistic?
Realistic paper trading models order states, partial fills, rejections, fees, latency and position updates, not only market prices. Its fill assumptions should be documented and tested against a range of plausible conditions.
Is a broker sandbox enough to test a trading bot?
A sandbox is useful for API integration, authentication and basic order-state handling. It often simplifies matching, queue position and costs, so it should be paired with a simulator or other execution tests.
Can shadow trading prove that a limit order would fill?
No. Shadow trading can compare an intended order with live prices and trades, but it does not place the order in the queue. It can estimate possible fills, but those estimates depend on data and assumptions.
How should a trading simulator model limit orders?
Use available trade or order-book data to estimate whether the price traded and how much volume was available. Avoid assuming every touched limit fills, and test results across different queue, latency and slippage assumptions.
When should an agent move from paper trading to live orders?
Move only after repeated tests cover rejects, partial fills, cancels, retries, restarts and position reconciliation. Begin with strict size and loss limits, then compare actual execution records with the paper model before increasing exposure.
