Week one passes. Week six finds the gap.
Week one: an agent gets API credentials, reads a quote, submits a market order and sees a fill. The sandbox balance updates, the position matches the order and the integration test turns green. Week six: a live order receives a partial fill, a cancel request times out, a retry lands twice, and the agent's position no longer matches the broker's.
That is the useful timeline. Broker sandboxes make basic request and response flows easy to verify.
They often make market mechanics, operational failure and the cost of being wrong look simpler than they are. "Lie" describes the impression, not necessarily the broker's intent. A sandbox can accurately test authentication and payload format while saying very little about live execution.
For software that trades unattended, a successful sandbox order is evidence that the API accepted a request under simulated conditions. It is not evidence that a live order would fill at the displayed price, that a cancellation would beat an execution, or that the agent can recover from an ambiguous response.
The gap matters because agents turn small assumptions into repeated orders. A one-basis-point cost error on a $100,000 notional trade is $10 per side. A duplicate retry can cost much more, and a stale position can cause the next order to compound the mistake.
What the sandbox usually leaves out
Most broker sandboxes are designed to help developers integrate. They provide predictable balances, familiar symbols and a fast path from request to response.
That is useful for checking schemas and basic state handling. It becomes misleading when developers treat those convenient defaults as execution evidence.
Common omissions include queue priority, displayed depth, market impact, trading halts, partial fills and order-book changes between quote and execution. A market order may fill immediately at the last quote, even though a live order would sweep several price levels. A limit order may fill simply because the simulated price touched its limit, with no representation of the quantity already ahead in the queue.
Lifecycle behaviour is another weak spot. A live order can pass through accepted, open, partially filled, pending cancel, cancelled or rejected states.
A cancel request is not the same as a cancellation. If the order fills while cancellation is in flight, the agent needs to process the fill, not assume that its request won the race. Sandbox event streams may simplify this sequence or deliver events in a cleaner order than a networked system does under load.
Test these mechanics explicitly rather than inferring them from a green demo:
- Submit the same client order ID twice, then repeat after a client timeout. Confirm whether the broker deduplicates it and how the agent detects an uncertain outcome.
- Cancel an order immediately after submission and handle both possible results: cancellation succeeds, or the order fills before cancellation takes effect.
- Restart the agent after a partial fill. Reconcile open orders, executions and positions before sending anything new.
- Feed delayed or missing market data. Block new risk when the quote is stale rather than treating the last price as current.
- Exercise rejected orders, rate limits, session boundaries and reconnects. Confirm that retries use bounded backoff and do not multiply exposure.
Broker behaviour also differs by product. Equity orders face opening and closing auctions, halts, corporate actions and short-sale constraints.
FX prices can widen around rollover and news. Options have contract-specific liquidity and spreads. A single "paper fill" rule is not a useful substitute for those differences.
Price the missing mechanics
The table gives typical market ranges for common instruments and API paths. These are working estimates, not audited measurements of any named broker.
Actual costs depend on venue, order size, time of day, account schedule and market conditions. The sandbox column describes common simplifications, not a guarantee about every provider.
| Cost or mechanic | Typical sandbox behaviour | Typical market ranges | Budget and test implication |
|---|---|---|---|
| US equity spread | Fill near the quote or last trade | About 1 to 5 basis points for liquid large-cap shares; 10 to 100 or more for less liquid shares | Model bid-ask cost on entry and exit. Test by liquidity bucket. |
| FX spread | Stable reference price, little widening | About 0.1 to 1.5 pips for major pairs in ordinary conditions; 2 to 10 or more around rollover or news | Test time-of-day widening and reject or spread limits. |
| Options spread and fees | Full fill at a selected price, often without realistic queueing | Spreads can run from 1 to 10 or more ticks; commissions are commonly about $0.50 to $0.75 per contract per side, plus applicable fees | Include spread, contract fees and partial fills. Do not assume mid-price execution. |
| Marketable-order slippage | Little or none beyond the displayed price | About 0 to 20 or more basis points for modest orders in liquid shares; larger orders can exceed this | Size against available depth and stress orders that exceed it. |
| API acknowledgement | Near-instant response in a stable test environment | Roughly 50 to 500 ms on a typical internet path, with longer tails during congestion or incidents | Set timeouts for the tail, not just the median. Treat timeout as an unknown order state. |
These figures make a difference to sizing. Suppose an agent turns over $100,000 of liquid US shares each day.
A one-way spread and slippage allowance of 2 to 8 basis points implies roughly $20 to $80 per side, before commissions and other fees. If the strategy expects 5 basis points of gross edge, a simulator that assumes free fills has removed the main constraint on whether the strategy works.
There are also costs that do not fit neatly into a per-trade estimate. A missed cancel can leave unwanted exposure. A delayed position update can cause an oversized replacement order. A broken market-data stream can keep a strategy trading on stale inputs. Put these into the risk budget as failure scenarios, with explicit limits on notional, order count and time spent in an uncertain state.
A sandbox fill proves that a simulated order filled. It does not prove the live market owed the agent that price.
Build a test budget, not a fill-rate target
Count what the simulation omits, then decide how to approximate each omission. A useful test plan is specific enough to fail on a known defect. "The strategy made money in paper trading" is not a test of order handling or realistic costs.
- Record the assumptions. Write down how the simulator prices market orders, decides limit fills, handles partial quantity, applies fees and advances positions. If a behaviour is undocumented, mark it unknown rather than treating it as realistic.
- Set a cost envelope. Add spread, commission and slippage separately. Run a base case and adverse cases, for example twice the ordinary spread, a partial fill followed by a worse-price remainder, and a marketable order with 10 to 20 basis points of slippage. Use ranges suited to the instrument, not one haircut for every asset.
- Inject order-state failures. Simulate delayed acknowledgements, dropped events, duplicate messages, rate-limit responses and reconnects. Check that the agent queries broker state before retrying an order whose outcome is unknown.
- Measure exposure errors. Compare the agent's internal positions and open orders with a broker snapshot after each failure. Set a maximum tolerated mismatch, such as zero unexplained open orders and a small, defined position difference, before the system can resume trading.
- Replay and review. Store requests, responses, timestamps, quotes and execution events. Replay the same sequence after code changes. Differences in order count, fill assumptions or final exposure should be explainable.
A dedicated agentic paper-trading environment can help make scenario runs repeatable, but the label alone does not establish fidelity. Inspect the execution rules.
Check whether the system models partial fills and order-state races, whether market data can be delayed, and whether a run can be reproduced from recorded inputs. If those details are absent, treat it as an integration tool, not a live-market benchmark.
Allocate engineering time as well as compute or subscription costs. A small agent team might spend one to three days building a basic failure harness, then several hours each week maintaining scenarios as broker APIs and strategies change. That is usually cheaper than diagnosing a position mismatch after live orders have accumulated.
Promote only after recovery works
A better sandbox is not a substitute for controlled live deployment. It is one stage in a sequence.
The production question is not whether the agent can place an order. It is whether it can limit damage when the broker's response, market data or its own state is incomplete.
Before enabling unattended routing, require the agent to:
- Use a unique client order ID and a durable record of every submission attempt.
- Reconcile orders, executions and positions at startup, after reconnects and on a fixed schedule.
- Stop opening new risk if data is stale, account state cannot be reconciled, or a broker response leaves the order outcome uncertain.
- Apply limits to order size, gross and net exposure, daily loss, order rate and repeated rejects.
- Raise an alert and enter a defined safe state when a kill condition trips. Do not rely on a process crash as the risk control.
Use a small live pilot only after these conditions pass in simulation. Keep notional low enough that the worst plausible reconciliation error is tolerable.
Compare the live order lifecycle and effective execution cost with the test assumptions, then adjust the model. A few dozen trades can reveal gross mismatches, but they cannot establish stable slippage estimates across instruments, sessions and volatility regimes.
Broker sandboxes remain valuable for what they do well: authentication, request formats, account endpoints and basic integration. Their weakness is not that they are useless.
It is that a frictionless order response looks like a market result when it is only a software response. In this industry, a green status often means the easiest part passed.
Frequently Asked Questions
What does a broker sandbox actually test?
Usually it tests authentication, request formats and basic account or order endpoints. It may not model queue position, market impact, realistic partial fills or operational failures, so a successful sandbox order does not establish live execution quality.
How should I model slippage in paper trading?
Model spread, fees and slippage separately, using ranges suited to the instrument and order size. Run ordinary and adverse cases, then compare those assumptions with live pilot executions before increasing exposure.
Can I use broker sandbox fills to validate a trading strategy?
Use them to validate integration and basic order logic, not as sole evidence of strategy performance. Fill assumptions can materially overstate returns, especially for limit orders, less liquid instruments and strategies with small expected edges.
