A sandbox is a contract test, not a market replica
"Can we trust the broker sandbox to prove this agent is safe to run unattended?" No. It can show that your software can authenticate, format requests and handle some responses. It cannot prove that live orders will fill as expected, that account restrictions will match, or that the system will recover safely from a real execution failure.
Broker sandboxes are useful, but their purpose is usually to simulate an API contract or provide a test account. They may use delayed or synthetic prices, simplified matching and account state that resets independently of production.
A successful response means the sandbox accepted a request under its own rules. It is not evidence that a live venue would accept or fill the same order.
This distinction matters for compliance as well as execution. An unattended agent can create exposure through duplicate orders, stale position data or a misread rejection.
A test setup that always accepts requests can hide whether controls detect those conditions, stop the strategy and leave a usable record for review. Treat sandbox behaviour as one input to release approval, not as approval itself.
Document what the environment actually simulates: order types, trading hours, price source, margin model, account resets, rate limits and available asset classes. If a feature is undocumented, assume it is not tested. Keep the sandbox endpoint, credentials and account identifier separate from live configuration, and make the application refuse to start if its environment cannot be identified.
The risk gap sits in fills, state and permissions
The largest mismatch is often the fill model. A sandbox may fill a market order immediately at the displayed price. A live marketable order can cross a spread, move through multiple price levels or receive a partial fill. A limit order may appear filled when the price touches its limit in a simulator, although the live order would sit behind other orders in the queue and receive no fill.
Account state creates another gap. Production may apply buying-power checks, short-sale restrictions, borrow availability, trading halts, minimum order sizes or session rules that the sandbox does not reproduce.
Even when those checks exist, test values may not resemble the live account. An agent that calculates quantity from a stale balance can submit an order that is valid in testing and rejected in production.
Permissions deserve separate verification. A test token may expose endpoints or account actions that the live token does not, or vice versa.
Check the live account's actual permissions, including whether the credential can trade, withdraw, change settings or access multiple accounts. Use the narrowest available scope. Store credentials outside strategy code and make rotation and revocation part of the incident procedure.
A green sandbox response proves the request passed one simulation. It does not prove the risk controls will work under live conditions.
Build tests around failure states, not just successful order submission. At minimum, make the agent handle:
- Partial fills followed by a cancellation or delayed remaining quantity.
- Timeouts where the broker may have accepted the order but the client has not received the response.
- Rejections for buying power, trading hours, price bands or unavailable borrow.
- Stale quotes, delayed account updates and a disconnected stream while orders remain open.
- Duplicate submissions after a retry, process restart or repeated event delivery.
Use typical ranges as test assumptions, not promises
The ranges below are broad practitioner assumptions for comparing test and live behaviour across retail and API-accessible markets. They are not audited results for any named broker.
Asset, venue, order size, session and network path can move the figures substantially. Replace them with measurements from the specific route and account before setting limits.
| Measure | Typical range in a sandbox | Typical range in live trading | Risk to test |
|---|---|---|---|
| Order acknowledgement | 10-200 ms | 30-500 ms | Timeouts and late responses can leave order status uncertain. |
| Marketable order completion | 0-100 ms, often simulated as one fill | 10 ms-2 s, with partial fills possible | Position and remaining quantity can differ from the submitted amount. |
| Limit order at the quoted price | Often fills at a price touch | No guaranteed fill; queue waits can range from milliseconds to hours | A price touch does not establish queue priority or execution. |
| Slippage on liquid instruments | Often modelled as 0-0.1 basis points | Roughly 0-5+ basis points in ordinary conditions, potentially wider in thin markets | Zero-cost tests can make an uneconomic strategy appear viable. |
Do not turn these ranges into universal service-level targets. Record timestamps at request creation, broker acknowledgement, execution report and position update.
Compare client-observed times with broker event times where available. Track distributions, not just averages: a 99th-percentile delay of several seconds can matter more than a low median if the strategy relies on short-lived prices.
For repeatable runs outside a broker's own simulator, Agentic Trading Index agentic paper trading is available from $9.99 a month. Treat any paper environment as a way to exercise agent logic and controls, not as proof of live queue position, liquidity or slippage.
Turn mismatches into controls and evidence
Before enabling live credentials, make the application enforce rules independently of the strategy. A strategy should propose an order; a separate risk layer should validate quantity, instrument, price deviation, exposure and session.
The layer should reject orders when inputs are stale or the account state is unknown. It should not silently replace a rejected order with a larger or more aggressive one.
Use a staged release process. A small live canary can expose route-specific behaviour that neither sandbox nor paper trading reproduces, but it still needs explicit limits and monitoring. Decide the permitted notional, number of open orders, loss threshold, price collar and stop conditions before the run. Define who or what can disable trading, and confirm that the kill path cancels open orders where appropriate rather than only stopping new submissions.
- Keep a versioned record of strategy parameters, code hash, environment, account permissions and risk limits for each release.
- Replay the same order scenarios in the sandbox, paper environment and controlled live canary. Record differences in acknowledgements, fills, fees, rejections and resulting positions.
- Set explicit acceptance criteria. For example, require every submitted order to reconcile to a broker order identifier or a documented terminal error within a chosen time limit.
- Alert on unresolved order state, unexplained position differences, repeated rejects and limit breaches. Make the default response to uncertainty a pause, not another order.
- Retain request, response, market data and decision records with timestamps and correlation identifiers. Apply retention and access rules that match the relevant jurisdiction, firm policy and instrument.
Keep compliance evidence separate from a strategy's own logs where possible. Capture the input data, risk checks, order intent, broker response and any operator action so an investigation can reconstruct what happened. Obligations differ by jurisdiction, product and firm status, so map the records and controls to the applicable rules rather than assuming that a sandbox's logs are sufficient.
Make release approval evidence-based
A sandbox is most useful when its limits are explicit. Maintain a mismatch register that names each unsupported feature, the consequence if it behaves differently and the compensating control. For example, if the simulator cannot model queue position, strategy approval should not rely on assumed fills at touch; use conservative fill assumptions and validate performance against live observations before raising limits.
Review that register after API changes, account migrations and broker notices. A change to order status semantics or rate limits can invalidate tests without changing your agent code.
Monitor live rejections and fill differences against the baseline, and pause deployment when behaviour moves outside pre-agreed bounds. Keep the pause condition machine-readable so it does not depend on someone interpreting a dashboard alert.
If this were my operation, I would first inventory every sandbox assumption and mark each one that could change exposure, permissions or order state. Then I would run a small, tightly capped live canary with reconciliation and automatic pause conditions before allowing the agent to trade unattended at normal limits.
Frequently Asked Questions
What does a broker sandbox actually test?
It usually tests authentication, request formatting and selected API responses under simulated account and market conditions. It does not establish live fill quality, queue position or production risk checks unless those behaviours are explicitly reproduced and verified.
Can a sandbox fill differ from a live broker fill?
Yes. Simulators may fill at a quoted price or immediately, while live orders can be partially filled, rejected or left waiting in a queue. Test these outcomes in your order-state handling and compare them with measured live observations.
How should a trading agent handle an uncertain order response?
Do not blindly resubmit after a timeout, because the broker may have accepted the original order. Reconcile using a client order identifier or broker query, then pause new exposure if the order state cannot be established within a defined limit.
