The order was accepted. That proves the request passed a software interface, not that the broker could execute it at the quoted price, in the expected quantity, or before the market moved. A sandbox can look healthy while an unattended agent is learning the wrong execution model.
Most broker sandboxes are not deliberately deceptive. They answer a narrower question than developers often assume. They test request formats, authentication and some state transitions, then simulate fills using simplified rules. The gap is measurable, but only if each test defines what an order outcome means and records the evidence needed to compare it with live execution.
For an agent, the practical risk is not just a few basis points of slippage. A simulator that fills every limit order can teach a strategy to submit orders that would sit unfilled, miss the move and leave inventory out of balance. A simulator that treats an acknowledgement as a fill can produce incorrect position limits and duplicate exposure after retries.
Acceptance is not execution
Separate the order lifecycle into events: client submission, broker receipt, acknowledgement, working state, partial fill, full fill, cancellation request, cancellation confirmation, rejection and expiry. Store the broker event time and the time your system received each event.
Those clocks answer different questions. A delayed callback can make an order appear late even when execution happened promptly.
Broker APIs also use these terms inconsistently. An HTTP success may mean only that the request was parsed.
A returned order ID may mean it entered a queue, not that it is working at a venue. Read the API contract, then test the interpretation. Treat every state transition as provisional until an execution report or equivalent authoritative event confirms it.
Use stable client order identifiers and persist them before sending. On a timeout, query the order by that identifier before resubmitting.
A sandbox that always replies quickly will not expose the dangerous case: the broker accepted the first request, the reply was lost, and a retry creates a second live order. Record duplicates as a separate outcome, even when your client eventually detects and cancels them.
For each attempt, retain the instrument, side, order type, limit price, requested quantity, decision timestamp, quote snapshot, submission and response timestamps, every broker event, and the final filled quantity and average price. This dataset should be append-only. Reconstructing the lifecycle from a single final status discards the evidence needed to diagnose race conditions.
Choose metrics with explicit denominators
Use a small set of measures with definitions that survive a change of broker or simulator. Do not compare a percentage of requests in one environment with a percentage of orders in another. Report the raw counts alongside each rate, and split results by instrument, order type, session and market condition.
- Fill ratio: executed quantity divided by submitted quantity. Report both per-order completion and aggregate quantity completion. A hundred orders filled for one share each do not represent the same outcome as one hundred fully filled orders.
- Reject rate: broker-rejected valid submissions divided by valid submissions. Keep client-side validation failures separate. Break rejection reasons into categories such as price bands, insufficient buying power, market status, unsupported order type and rate limits.
- Execution shortfall: compare fill price with a recorded reference price. For a buy, a positive cost in basis points is (fill price minus reference price) divided by reference price, multiplied by 10,000. Reverse the sign for a sell so positive values consistently mean worse execution.
- Latency: measure client-to-acknowledgement, acknowledgement-to-first-fill, cancel-request-to-confirmation and event-occurrence-to-client-receipt separately. Report median, 95th and 99th percentiles. A mean hides the long delays that can break an agent's state.
- Cancel and partial-fill exposure: record the quantity filled between a cancel request and confirmation, plus the time an order remains partly filled. A successful cancel response does not undo executions that occurred before cancellation took effect.
Keep two shortfall references. The decision-time midpoint captures the combined cost of decision delay and execution. The midpoint at broker receipt is a closer measure of execution quality after the order reaches the broker. Include spread and fees separately where possible; otherwise, a favourable price movement can make a costly fill look good.
Set a benchmark before comparing environments
The ranges below are illustrative typical market ranges for planning tests, not broker guarantees or audited measurements of any named company. Instrument liquidity, market hours, order type, API design and volatility can move outcomes well outside them.
Sandbox figures are especially dependent on simulator rules. Their purpose is to show which measurements commonly diverge, not to certify a platform.
| Measure | Sandbox typical range | Live typical range | What to test |
|---|---|---|---|
| Acknowledgement latency | 5-100 ms | 20-500 ms | Tail latency, timeouts and duplicate submissions |
| Marketable order to first fill | 0-100 ms | 50 ms-2 s | Price movement and partial execution |
| Limit-order fill ratio | 80-100% | 20-90% | Queue position, touch versus trade-through rules |
| Reject rate in ordinary conditions | 0-0.1% | 0.1-2% | Buying power, price bands, session and throttles |
| Cancel confirmation latency | 5-100 ms | 50 ms-1 s | Fills arriving during the cancel race |
| Shortfall on liquid instruments | 0-2 bps | 0.5-10 bps | Spread, fees, size and changing liquidity |
Do not use the table as a pass-or-fail score. A sandbox can have slower acknowledgements than live trading and still be useful. The important question is whether it models the failure modes that affect the strategy.
For a passive strategy, queue behaviour and unfilled orders matter more than a fast simulated acknowledgement. For an urgent risk-reducing order, cancellation and marketable-order execution may dominate.
Where market data is available, estimate whether a limit order would have filled using trades and quotes after submission, not merely because the best price touched the limit. A touch does not prove the order had queue priority. If the simulator cannot model queue position, label fills as optimistic and do not treat its fill ratio as a forecast.
Design tests that expose the gap
Run the same deterministic order scenarios in both environments where access permits. Fix the instrument, side, quantity, order type and timing. Save the input quote and the full event stream.
Compare distributions, not one successful run. For live tests, use only authorised, appropriately sized orders and account for fees, market rules and operational controls.
Include normal conditions and deliberate faults. Inject a client timeout after submission, delay event delivery, restart the process with an open order, and send a cancel while a partial fill is possible. Test market open and close separately from quiet hours. If the broker restricts sandbox traffic or does not simulate a condition, mark that coverage as unknown rather than passing it by default.
- Submit a marketable limit order, then compare requested quantity, first-fill time, final quantity and shortfall against the receipt-time midpoint.
- Place a passive limit order at the bid or offer. Check whether the simulator fills on a touch, a trade through the price, or an explicit queue rule.
- Cancel during a partial fill. Confirm that filled quantity remains booked and that the agent does not assume the entire order was cancelled.
- Drop the submission response, restart the client, and reconcile by client order ID before allowing a retry.
- Trigger buying-power, price-band, closed-market and rate-limit conditions. Check that the rejection reason reaches the agent and its risk state.
For each scenario, record expected behaviour before running it. Measure the percentage of runs with the correct final position, not just the percentage with an API response.
A run is wrong if the agent reports flat while a fill remains open, even if every request returned a syntactically valid response. This position reconciliation metric is often more useful than a single success rate.
At least 100 repetitions can help expose ordinary variability, but that sample is weak for rare events. If no failure appears in 100 independent trials, the rough 95% upper bound on the true failure rate is still about 3%.
A test of 1,000 clean trials narrows that bound to about 0.3%. Treat repeated trials as independent only when the conditions really vary; replaying one static quote does not test market variability.
Make sandbox limits part of the release gate
Translate each limitation into a control. If fills are optimistic, cap size and require live execution monitoring before scaling. If cancel races are not simulated, reconcile open orders after every cancel and block replacement orders until the broker confirms state. If rate limits are unknown, use conservative request budgets and backoff. If the event stream can reorder messages, make state updates idempotent and reconcile against broker snapshots.
Paper trading can test the agent's own decision loop, accounting and recovery logic without risking capital. It cannot establish that a particular live route will fill at a particular price.
Agentic Trading Index offers agentic paper trading from $9.99 a month through its paper trading platform. Use it to measure your system, but keep the simulator's assumptions visible alongside every result.
Set release thresholds by strategy and failure cost. A market-making agent may need tight controls on stale quotes and inventory drift. A low-frequency agent may accept higher callback latency but still require exact position reconciliation.
Define the maximum tolerable duplicate-order rate, unaccounted quantity, and time spent with uncertain order state. If the sandbox cannot exercise a condition, require a separate operational safeguard rather than marking the condition tested.
Keep a versioned record of simulator behaviour. Re-run the suite after API, client library, strategy or risk-rule changes, and when broker documentation or sandbox behaviour changes.
Track the timestamp, test inputs, response schema and observed state transitions. This turns a vague claim that paper trading works into a bounded statement about what has and has not been tested.
The most revealing sandbox score is not how many requests it accepts. It is how often the agent ends with the correct broker-confirmed position after delays, partial fills, rejected orders and lost responses. Check that rate in your own test suite, with enough trials to make the number meaningful: what is your observed position-reconciliation failure rate across the last 1,000 order lifecycles?
Frequently Asked Questions
What does a broker sandbox actually test?
It usually tests authentication, request formats and a subset of order-state handling. Simulated fills may use simplified pricing and queue rules, so a successful response does not establish that a live order would fill.
How should I measure broker sandbox execution?
Track fill ratio, rejection rate, shortfall, latency percentiles, cancellation races and final position accuracy. Define each denominator and retain the complete order event stream so results can be reproduced.
What is a good order fill ratio in a sandbox?
There is no useful universal target because simulators apply different fill rules. Compare fill ratios by order type and price behaviour, and test whether the simulator fills passive orders on a mere touch.
How many trials are enough to test an order workflow?
Around 100 repetitions can reveal common issues, but not rare failures. With no failures in 100 independent trials, the rough 95% upper bound on the true failure rate is still about 3%; 1,000 clean trials reduce it to about 0.3%.
Can paper trading predict live slippage?
It can help test measurement and decision logic, but a simulator without live queue position, fees and venue conditions cannot reliably predict live slippage. Measure shortfall against both the decision-time and broker-receipt midpoint where data permits.
