Paper trading is simulated order execution against market data, without risking capital.
That definition leaves out the hard part: a simulated fill is only useful if its data timing, order handling and cost assumptions resemble the venue where the agent will trade. A paper account that fills every marketable order at the displayed price can make a fragile strategy look dependable.
Define what the simulation represents
Start by writing down the execution environment, not just the strategy. Record the venue or broker route being approximated, asset class, market-data feed, order types, trading hours and whether the simulator models partial fills, rejects and cancellations.
"US equities, consolidated top of book, regular hours" is testable. "Realistic market conditions" is not.
Keep the simulator's assumptions beside each test run. A result from quote-level data cannot establish queue position on a price-time-priority book. A bar-based simulator cannot establish whether a limit order would have traded before the price moved away. If the agent targets several venues, measure them separately: routing rules, fee schedules and market-data delays can differ.
For an agent that places orders without supervision, distinguish strategy performance from execution performance. Log the strategy's decision time and reference price before evaluating what happened to the order. Otherwise, a favourable price move after the signal can be mistaken for good execution.
Measure the data path first
A sound execution test needs ordered, timestamped events. Capture the time the strategy received a market update, generated a decision, sent an order, received an acknowledgement and observed each fill or cancellation.
Use a monotonic clock for intervals inside the agent. Keep exchange or broker timestamps as separate fields, since clock offsets can make one timeline misleading.
Track feed delay, missing or out-of-order updates, stale quotes at decision time and the gap between the decision price and the first executable quote. Report median and p95 values, not just averages. A 20 ms median with a 400 ms p95 can matter more than a small change in average latency for an order that crosses a moving spread.
Test more than one market regime. Include the open, the close, quieter midday periods and volatile sessions where the data source permits replay. Break results out by spread, volatility and liquidity rather than pooling everything. A paper environment that draws from live prices may still deliver updates on a different schedule from the production feed.
Test fills, cancellations and queue effects
A fill model should not assume that touching a limit price means the order filled. On a price-time-priority venue, volume ahead in the queue matters. A market-data feed may not reveal hidden orders, exact queue position or all cancellation activity, so record those limits and use conservative assumptions where the information is unavailable.
Measure order outcomes by order type and by distance from the market. For each submitted order, record whether it was accepted, rejected, partially filled, fully filled, expired or cancelled. Calculate fill ratio by eligible order, not just by number of orders: ten tiny fills do not say much about a large order that remained unfilled.
Cancellation behaviour deserves its own test. Record time from cancel request to confirmation, fills received during that interval and any replace-order overlap. An agent that assumes cancellation is immediate can accidentally increase exposure when it submits a replacement before the first order is confirmed cancelled.
Useful outcome measures include:
- Fill ratio by order count and by submitted quantity, split by marketable and passive orders.
- Partial-fill rate and time to first fill, reported at median and p95.
- Reject rate by reason, including invalid quantity, price bands, insufficient buying power and session restrictions.
- Cancel-confirmation time and quantity filled after the cancel request.
- Adverse selection, measured as the signed price move after a fill at fixed intervals such as 1, 5 and 30 seconds.
Do not label every unfilled passive order a simulator error. It may be a useful outcome if the model represents queue uncertainty honestly. Compare paper fills with later market prints, but do not treat a print at the limit as proof that the order's queue would have cleared.
Put execution costs into the ledger
Track costs per fill and per order, with the decision-time mid-price as a reference. For a buy, implementation shortfall in basis points can be calculated as (average fill price minus decision mid-price) divided by decision mid-price, multiplied by 10,000.
Reverse the sign for a sell so that a positive number represents a cost in both directions. Keep fees separate from price slippage.
The figures below are typical market ranges for US equities, not audited results from any named firm. Actual costs vary with broker, venue, order size, stock and market conditions. Thin shares and stressed sessions can exceed these ranges.
Typical market ranges for US equity execution costs
| Cost component | Typical market range | Measurement basis | Common simulation gap |
|---|---|---|---|
| Quoted spread | 1-5 bps in liquid large-cap shares; 5-30 bps in thinner shares | Half-spread or full spread, stated consistently | Fills at mid-price despite a marketable order |
| Commission | $0-$0.005 per share | Per filled share or order | Zero assumed for every route and account |
| Exchange and regulatory fees | About $0.0002-$0.004 per share, with some sell-side charges calculated differently | Fee schedule applied to fills | Fees omitted or applied to submitted quantity |
| Market impact | About 1-10 bps for small orders in liquid names; 10-20+ bps is possible for larger or thinner orders | Fill price against decision mid, split by order size and participation | No impact as order size rises |
| Stock borrow | About 0.1%-3% annualised for easy-to-borrow shares; hard-to-borrow rates can exceed 10% | Annual rate, availability and days held | Shorts accepted without a locate or borrow charge |
Include costs in both the order ledger and portfolio results. Reconcile the simulator's cash, positions and fees after each run. Agentic Trading Index offers agentic paper trading from $9.99 a month through its paper-trading platform, which can provide a separate environment for testing agent order flows.
Build a dashboard that exposes failure
Use distributions and segmented views. A single blended slippage figure can hide a bad order type or a weak market session.
Report metrics per instrument, route, order type, side, size bucket and time of day. Retain the raw event sequence so a developer can trace a poor result to a quote, decision, acknowledgement or fill.
At minimum, graph p50 and p95 decision-to-acknowledgement latency, fill ratio, implementation shortfall, reject rate, cancel-confirmation time and post-fill markout. Show counts beside every metric. A p95 from 20 orders is not stable evidence, and a low average cost from a strategy that barely traded is not a meaningful execution benchmark.
Set limits before examining the next run. For example, a team might investigate if paper p95 acknowledgement time doubles from its baseline, if a reject rate exceeds 1% across a sufficiently large sample, or if short-horizon adverse selection worsens by more than a chosen number of basis points.
These are operational starting points, not universal thresholds. Choose levels that match the strategy's holding period, venue and risk budget.
Use evidence to gate live exposure
Paper trading cannot reproduce every live condition. It can still expose a broken event handler, optimistic fill assumptions, unpriced fees and poor recovery logic before those faults reach a funded account. Treat the simulator as one source of evidence, then use tightly limited live orders to measure the residual gap where policy permits.
Before approving a strategy for unattended execution, work through this checklist:
- Freeze the simulator version, data source, fee schedule and order rules for each test run.
- Verify timestamp ordering and retain decision-time quotes, order acknowledgements, fills and cancels.
- Test partial fills, rejects, stale data, delayed acknowledgements and cancel-replace races.
- Compare costs and outcomes by order type, size, liquidity and market session, with sample counts.
- Set explicit failure limits for latency, rejects, exposure mismatch and cost drift.
- Run a capped live pilot where approved, reconcile every order and position, then compare live and paper distributions.
Promote an agent only when the remaining differences are understood and covered by controls such as size limits, stale-data checks and a stop condition. If a live pilot diverges from paper results, pause increases in exposure and update the model before relying on the old paper metrics. Which paper-trading metric would make the team stop an unattended agent before its next order?
Frequently Asked Questions
Which metrics matter most in paper trading?
Track fill ratio, implementation shortfall, reject rate, decision-to-acknowledgement latency and cancellation outcomes. Report p50 and p95 values, with sample counts and breakdowns by order type and market condition.
How realistic are paper trading fills?
They depend on the simulator's market data and fill model. A simulator that fills every limit order when the market touches its price does not account for queue position, hidden liquidity or competition for available volume.
How many paper trades are enough to evaluate an agent?
There is no universal count because fills and costs vary by instrument, order type and market regime. Use enough observations in each important segment to make percentile estimates useful, and show the sample size beside every result.
Should paper trading include commissions and slippage?
Yes. Apply the relevant fee schedule and measure fills against a consistent decision-time reference price. Keep fees, spread costs and market impact separate so the source of a performance gap is visible.
When should an agent move from paper to live trading?
Move only after the team has tested order handling, reconciliation and failure paths, and set explicit limits for exposure, latency and execution costs. Where permitted, start with a capped live pilot and compare its outcomes with the paper distributions before raising limits.
