At the end of a bad week, a developer opens the dashboard and sees a steady equity line. The broker statement shows three rejected orders, a stale quote before a large fill and a position that differs from the agent's records.
The dashboard was healthy. The trading system was not.
Monitoring an unattended agent means measuring whether its decisions were based on current data, whether orders reached the venue as intended, and whether the live account still matches the software's view. A process running without exceptions is only one small part of that picture.
Give every measurement a clock
Trading metrics need timestamps that describe different stages, not one generic time field. A market-data event has an exchange or vendor timestamp and a local receive timestamp. An order has a decision time, send time, broker acknowledgement time, fill time and any cancellation or correction time. Store them all in UTC, along with a monotonic clock for measuring elapsed time inside the process.
That distinction makes latency interpretable. If an order acknowledgement takes 700 milliseconds, the cause could be a slow network, a busy broker API or time spent building the request. If the agent only records the time the order function returned, those causes collapse into one number. Record both the total duration and the relevant spans.
Attach a stable strategy, account, instrument and client order identifier to each event. Keep those dimensions in logs and traces, but avoid using them all as metric labels. High-cardinality labels can make a metrics store expensive or slow. Aggregate counters by strategy and venue; keep individual identifiers in searchable event records.
Use counters for events such as submissions, rejects and reconnects. Use histograms for latency and slippage, so a p95 or p99 can reveal a tail that an average hides.
A one-minute average can look acceptable while a small share of orders waits several seconds. Choose aggregation windows to match the strategy: sub-second execution needs fine-grained views, while a slower rebalance may be well served by one-minute summaries.
Measure the order path, not just the process
The order path starts before the API call. Record how old the input quote was when the decision was made, the expected price, requested quantity, order type and any risk checks that altered or blocked the request. Then capture the broker response, execution reports, fees and final quantity. Without this chain, a fill that looks poor cannot be compared with the market state the agent actually observed.
The figures below are typical operating ranges for liquid instruments and common API-based setups, not universal limits or audited results for any provider. Venue, instrument, session, order size and feed design can move them materially. Establish a baseline from the agent's own paper and live observations, then set limits around the system's actual use case.
| Metric | Typical range | What to investigate |
|---|---|---|
| Quote age at decision | About 0.1 to 2 seconds | Repeated readings above the strategy's allowed age, especially during active sessions |
| Order acknowledgement latency, p95 | About 100 milliseconds to 1 second | A sudden increase from the venue and network baseline, or a long tail above several seconds |
| Rejected order share | About 0.1% to 1% in ordinary operation | Clusters by instrument, order type or reason code; a sustained rate above 1% deserves review |
| Adverse slippage on small liquid orders | Roughly 0.1 to 1 spread width | Repeated costs above the strategy's model, or a sharp increase in a particular session |
| Position difference after reconciliation | Zero units expected | Any unexplained difference, including a small residual that persists after pending fills settle |
Measure execution quality against a defined reference. For a marketable order, that might be the mid-price at decision time, with spread, fees and slippage reported separately.
For a passive order, track time resting, partial-fill rate, cancellation rate and the price movement after cancellation. A high fill rate is not automatically good if fills arrive mostly before the market moves against the position.
Keep metrics for requested and executed quantities distinct. A partially filled order is neither a full success nor a simple failure. Report fill ratio by order and in aggregate, and preserve the remainder's status. If a cancel request times out, the order may still be live. The monitor must keep checking for subsequent fills rather than treating the timeout as confirmation of cancellation.
Separate market loss from system failure
An agent can lose money while behaving exactly as designed. Monitoring should make that visible without labelling every losing interval as an incident.
Compare realised and unrealised P&L with exposure, turnover, fees and a strategy-specific risk budget. Break results down by instrument and session where the sample supports it. A daily loss figure without open risk can conceal a position that is still accumulating exposure.
Separate operational alerts from performance review. A disconnected feed, rejected risk-reducing order or unexplained increase in gross exposure may require immediate action. A negative 30-minute return may not. Thresholds should account for the strategy's holding period and volatility, and should combine a level with a persistence rule where a brief spike is harmless.
- Data health: quote age, missing sequence numbers, feed reconnects and the share of instruments with current prices.
- Order health: submission-to-acknowledgement latency, rejection reasons, unresolved orders, fill delays and cancellation outcomes.
- Risk state: gross and net exposure, concentration, use, open-order notional and distance to configured limits.
- Strategy behaviour: turnover, signal distribution, expected versus realised trade count and P&L after fees.
- Account state: buying power, cash, margin utilisation, fees and broker-reported positions.
Compare live behaviour with a baseline, not only a fixed global number. An order latency of 500 milliseconds may be routine for one route and a serious regression for another. A rejection rate can rise temporarily during a volatile open, but repeated rejects for insufficient margin point to a sizing or state problem. Alert on both absolute safety limits and deviations from a rolling baseline, with a minimum event count so one unusual fill does not produce noisy statistics.
Paper trading helps test the instrumentation and expected event flow, although it cannot reproduce every live fill or venue failure. An agentic paper-trading environment can provide a controlled place to check that dashboards account for partial fills, delayed acknowledgements and restarts before the agent handles live orders.
Reconcile the account the broker actually holds
The agent's internal position is a calculation. The broker's position is the account state. Reconcile them on startup, after reconnects, after fills and at a regular interval. Compare positions, open orders, cash and buying power. Treat unexplained differences as a state problem until resolved, not as a harmless reporting discrepancy.
Reconciliation needs rules for timing. Broker updates can arrive out of order, and a position snapshot may be taken while executions are still streaming in. Use execution identifiers to deduplicate reports. Track the last sequence or update time where the API supports it. If it does not, compare snapshots only after a defined settling interval and retain the events used to explain each change.
Define what happens when the two views differ. The agent might pause new risk-increasing orders, query open orders again and rebuild its position from broker data.
It should not blindly resend an order because a local acknowledgement timed out. That can create a duplicate position if the original request reached the broker.
- On process start, fetch positions, balances and open orders before enabling strategy decisions.
- After a disconnect, request a fresh snapshot, replay available execution events and check for orders with unknown status.
- When a difference remains, record the size, age and account impact, then block actions that could increase the discrepancy.
- After recovery, log the reconciliation result and the source of truth used to restore local state.
Watch recovery time as a metric. A system that reconnects in one second but takes five minutes to restore a trustworthy position is not operationally recovered.
Track time to regain a valid market-data stream, time to resolve unknown orders and time until account state agrees. Set different service objectives for these stages rather than reporting a single green connection light.
Make alerts actionable without a person at the screen
An alert should state what failed, what evidence triggered it, what the agent did and what state remains uncertain. Include the account and strategy, affected instruments, relevant timestamps, current exposure and links to the underlying event trace.
Avoid alerts that say only that a function failed. That forces whoever investigates to reconstruct the event before making a safe decision.
Not every alert should stop trading. A stale quote on one symbol may justify suppressing that symbol while allowing others to continue. A lost account connection, position mismatch or unknown order status may require a broader pause. Make the response explicit in code and in the alert record. A notification is not a control unless the agent changes its behaviour or a person is expected to respond within a defined time.
Useful alert policies usually set different thresholds for warning, restriction and shutdown. They also use a cooldown or grouping rule to prevent a burst of related messages becoming a flood.
Keep a durable incident record even when notifications are grouped. If an automated recovery succeeds, record the initial fault, retry count, elapsed time and final state; otherwise a high recovery rate can hide a degrading service.
Test the monitoring path as deliberately as the order path. Inject stale data, delay acknowledgements, duplicate execution reports and interrupt the connection between a fill and local state update.
Confirm the right metric moves, the correct alert fires and the prescribed restriction takes effect. A paper environment can test these scenarios repeatedly, but monitoring still needs checks against live broker events and real network behaviour.
Every operational metric should have an owner, a threshold, a response and a retention period. Keep raw order and account events long enough to investigate the longest relevant holding and settlement periods, subject to privacy and regulatory requirements.
Dashboards are for diagnosis; alerts are for decisions. A green status is useful only when its checks cover the account, the data and the orders.
Trading has long measured returns more carefully than the machinery producing them.
Frequently Asked Questions
What should I monitor in an unattended trading agent?
Track market-data freshness, order latency, rejects, fills, cancellations, exposure, P&L after fees and broker-to-agent position differences. Keep event timestamps and identifiers so each metric can be traced to the underlying order or account update.
How often should a trading agent reconcile positions?
Reconcile at startup, after a connection recovery, after execution events and on a regular schedule. The interval depends on the strategy and API, but the agent should not enable new risk-increasing orders until it has a trustworthy view of positions and open orders.
What is a normal order rejection rate for an API trading system?
For ordinary operation, a rejection share around 0.1% to 1% is a useful typical range to investigate against, not a universal benchmark. Group rejects by reason, instrument and order type, since a small total rate can still conceal a persistent sizing or permission fault.
Should every trading system alert stop the agent?
No. A stale quote for one instrument may justify pausing that instrument, while an unknown order status or unexplained account position may require a wider pause. Define the action for each alert class in advance and record whether the agent applied it.
