A 1 basis point execution error on $1,000,000 of daily turnover costs $100 a day, or about $25,000 over 250 sessions. Two basis points doubles that, before financing, borrow costs or fees. Broker sandboxes can make this error invisible by filling at a quoted price, accepting every order and skipping the queue.
Their responses may be valid API responses. The simulated market behind them is often too forgiving for an unattended agent.
Find out what the sandbox is actually simulating
Most broker sandboxes are not deliberately deceptive. They simplify the parts that are difficult or expensive to reproduce: matching, market impact, routing, market data delays and account constraints. That simplification still misleads if the test result is treated as evidence of live execution quality.
Start by listing each assumption your strategy depends on. Does it use a market order, a limit order, or both?
Does it submit outside regular hours? Can it short, trade fractional quantities, or hold positions across a financing cutoff? Does it cancel and replace orders when a quote moves? Record the answer for the sandbox and for the live account configuration. An API with the same endpoint names can still have different market rules, permissions and order handling.
Collect timestamps and state changes, not just the final position. Log the request, client order ID, acknowledgement, every order update, each fill, the quote used by the strategy and the resulting account snapshot.
Use UTC timestamps and a monotonic clock for elapsed time. This makes it possible to tell whether an apparent instant fill was a simulation shortcut or a real response to a changing book.
Compare the assumptions with typical live ranges
The figures below are typical market ranges for liquid US equities, not promises about any broker. Instrument, order size, session, routing and volatility can move the live result well outside them. The point is to identify what the sandbox does not model and design a probe for it.
| Measure | Sandbox, typical range | Live, typical range | What to record |
|---|---|---|---|
| Order acknowledgement | 5-100 ms | 20-500 ms, with longer tails | Median and p99 request-to-ack time |
| Spread on a liquid share | 0-1 bp, often fixed | 1-5 bp in ordinary conditions, wider when stressed | Bid, ask and midpoint at decision and fill |
| Slippage on a small marketable order | Often 0 bp or deterministic | 0-5 bp in ordinary conditions, 5-50+ bp when illiquid or stressed | Fill price against the decision-time midpoint |
| Explicit commission | Often $0 in simulation | $0-$0.005 per share is a typical retail range, plus possible pass-through fees | Commission and fees from the execution report |
A full fill at the displayed quote does not show that a live order would have reached the front of the queue. A simulated cancellation that completes immediately does not show whether the original order could fill while the cancel is in flight.
Test order-state transitions as well as prices: accepted, partially filled, cancelled, rejected and expired. Verify how the API represents each state and whether duplicate updates are possible.
A sandbox validates your request format. It does not prove that the market will give you the simulated fill.
Run a five-day test this week
Keep the probe small and repeatable. Use the same symbols, quantities, order types, session times and client logic in the sandbox and, where permitted, a tightly capped live test. Do not compare one quiet sandbox session with a volatile live session and call the difference a broker effect. Save the raw event stream so results can be recomputed.
- Day one: freeze the setup. Write down account permissions, product, currency, market session, order types, price increments and data source. Capture the sandbox and live API versions, configuration and rate limits. Set a maximum order size and a maximum daily loss for any live probe.
- Day two: probe order behaviour. Submit small limit orders away from the market, then test a marketable limit order in a liquid instrument. Exercise cancel, replace and expiry paths. Do not leave unattended orders open after the test window.
- Day three: measure execution. Compare acknowledgement latency, fill latency, partial-fill frequency, spread paid and slippage against the decision-time midpoint. Separate marketable orders from passive limits. Report median, p95 and p99, plus the number of observations.
- Day four: test failure handling. Exercise invalid quantities, insufficient buying power, closed-market submissions, rate limits and a deliberately interrupted connection. Confirm that retries use the same client order ID where supported, and that the agent checks order status before resubmitting. Track rejects and ambiguous timeouts separately.
- Day five: reconcile and set gates. Compare order events with positions, cash, fees and executions at the end of the session. Set measurable release limits for slippage, unexplained state changes, stale data and reconciliation differences. If a limit is breached, block new orders and alert an operator.
For repeated runs, use the same event logging and controls in a controlled agentic paper-trading setup. Paper trading is useful for testing strategy wiring and recovery logic. It remains a simulation, not a substitute for small, authorised live probes. A typical service price can start at $9.99 a month; the useful question is whether it lets the agent exercise the failure paths the strategy actually needs.
Make the gap part of the release decision
Do not collapse all differences into one average. A low mean slippage can hide a handful of costly tail fills. Break results out by instrument, order type, session and volatility.
For each group, keep the sample count and the worst observed result. Small samples are a warning about uncertainty, not proof that the worst case cannot happen.
Before increasing live size, compare the observed gap with the strategy's expected edge. If an approach expects 3 bp per trade and the live test shows 2 bp of typical spread and slippage, there is little room for fees, latency or adverse selection. Apply conservative costs in the simulator, cap order size, and retain a kill condition for abnormal rejects, stale quotes or position mismatches. A sandbox should help expose the code paths to test, not set the cost assumptions used to approve deployment.
- Stop treating a successful sandbox response as a successful market execution.
- Stop assuming zero slippage, instant cancellation or full fills.
- Stop retrying an ambiguous order before checking its state.
Frequently Asked Questions
What does a broker sandbox test?
It usually tests API authentication, request formats and simulated order-state responses. It may not reproduce live routing, queue position, spreads, market impact or account-specific constraints.
How can I compare sandbox fills with live execution?
Use the same instruments, order types, sizes and session conditions, then log quotes, acknowledgements, order updates and fills. Compare latency and fill prices against the decision-time midpoint, and keep separate results for different market conditions.
Should an automated trading agent place live orders after passing a sandbox test?
Not on that evidence alone. First test failure handling and reconcile executions, positions, cash and fees, then use authorised live probes with strict size and loss limits. Set explicit conditions that block new orders when state or data checks fail.
