Monitoring an unattended trading agent is the continuous verification that its data, decisions, orders and exposure remain within defined bounds.
That definition sounds like a dashboard problem. It is not. A process can be alive, its charts can look normal and its heartbeat can keep arriving while an order is stuck in an unknown state or the agent is trading on stale prices. Useful monitoring checks the chain of events, then takes a controlled action when that chain breaks.
Monitor invariants, not a wall of charts
Check what must remain true
Start with invariants: conditions that should hold regardless of strategy or market direction. Examples include no duplicate client order IDs, no position above the configured limit, and no live order without a corresponding local order record. These checks catch failures that a P&L chart cannot.
Separate liveness from correctness. A heartbeat proves that a process sent a message; it does not prove that its strategy loop is advancing or that it has consumed current market data. Emit distinct signals for:
- Process liveness, with a sequence number and timestamp.
- Market-data progress, including the latest exchange or vendor timestamp consumed.
- Decision progress, such as the latest completed strategy cycle.
- Execution progress, including order acknowledgements, fills and cancellations.
- Risk state, including current position, working quantity and limits.
Each signal needs an owner and a threshold. A process heartbeat every 5 seconds might be acceptable for a slower strategy; a feed-age limit of 5 seconds may be unacceptable for a strategy using short-lived quotes. Derive thresholds from the strategy's intended holding period and observed venue behaviour, not from a generic default.
Make status explain the failure
A single green or red status hides too much. Record the last successful step and the age of each dependency. If the last market update is 800 milliseconds old but the broker connection is healthy, the likely issue differs from a broker session that stopped acknowledging requests.
Use structured events with an agent ID, account, instrument, order ID, event time, receive time and sequence number. Keep both event time and receive time: the difference helps distinguish a late source event from a slow internal queue. Include a correlation ID across decision, order submission and broker response so an operator can reconstruct one trade without joining logs by guesswork.
Instrument the order path end to end
Monitoring should follow an intent through every state transition: strategy proposes, risk checks, order is submitted, broker acknowledges, venue accepts, partial fills arrive, and the remaining quantity is cancelled or filled. Persist each transition before treating it as complete. If the process restarts, it should recover from recorded state and reconcile with the broker rather than assuming its in-memory view is current.
Measure latency at each boundary. Record the time a decision is made, the request is sent, the acknowledgement is received and the fill is reported.
A single average hides the tail. Track median and high-percentile latency, such as p95 and p99, by venue and order type. A typical API acknowledgement may take tens of milliseconds to a few seconds, depending on broker, network and load. A rise in p99 can matter before the average moves.
Monitor the difference between local and broker state. Compare open orders and net positions on startup, after reconnect, after a timeout, and periodically during trading. A timed-out submission is ambiguous: the broker may have accepted it even if the client did not receive the acknowledgement.
Do not blindly resubmit. Query by client order ID or reconcile open orders first, then resolve the state using a documented rule.
Useful order-path checks include:
- Every submission has a unique client order ID and a durable local record.
- Every acknowledgement or rejection maps to a known submission, with unknown responses raised as an error.
- Partial fills update filled quantity without resetting the remaining quantity incorrectly.
- Cancel requests remain pending until a cancel confirmation or a fill resolves the order.
- Broker positions and local positions reconcile within a defined interval, including after reconnect.
Do not poll at the highest possible rate just to make a dashboard feel current. Broker APIs often apply request limits, and excessive polling can compete with order traffic. Use streaming events where they are reliable, then poll selectively for reconciliation. A paper environment can help test event handling and telemetry, but it may not reproduce live latency, throttling or disconnect patterns. Include those conditions in agentic paper trading tests and verify the behaviour against the intended live interface.
Set thresholds that lead to action
Use ranges as starting points
Thresholds are not universal. A liquid instrument on a low-latency connection has different tolerances from a less liquid contract or a strategy that trades only a few times a day.
The figures below are typical operating ranges seen in API-driven workflows, not guarantees or audited measurements. Set limits from measured baselines, instrument rules and the cost of a delayed response.
| Signal | Typical range | Monitoring response |
|---|---|---|
| Market-data age for active liquid instruments | About 0.5 to 2 seconds before warning; often 2 to 5 seconds before a stop condition | Block new decisions using stale inputs; cancel or manage existing orders under a separate rule |
| Broker acknowledgement | Often 0.1 to 2 seconds; investigate repeated delays above 3 to 5 seconds | Mark the order state unknown, then query or reconcile before retrying |
| Position reconciliation during active trading | Every 1 to 5 seconds for strategies needing tight control; slower for low-frequency use | Compare broker and local quantity; pause new risk if the mismatch persists |
| Process and strategy heartbeat | Typically every 1 to 10 seconds, with a missed-heartbeat limit of 2 to 3 intervals | Page or fail over only after checking whether the strategy loop, not just the process, is stalled |
| Risk limit utilisation | Warning around 70 to 90 per cent; hard limit at the configured maximum | Reduce or block new orders before the hard limit is breached |
Use a progression rather than one magic number. A warning can require persistence across several samples, while a hard breach can trigger immediately.
For instance, one delayed acknowledgement may be transient; repeated delays across 3 of 5 submissions could indicate a degraded connection. For hard constraints such as exceeding a position limit, waiting for a rolling average defeats the control.
Define what each severity means:
- Informational: record and trend, with no interruption to trading.
- Warning: notify an operator and constrain new risk if the condition persists.
- Critical: block new orders, cancel selected working orders or invoke a risk-reduction rule.
- Unknown state: stop actions that could duplicate exposure until reconciliation resolves the uncertainty.
Keep alert thresholds distinct from trip conditions. An alert asks someone to investigate. A trip condition tells software what to do without waiting for a person. State the action precisely: disable new entries, cancel unfilled entry orders, preserve exit orders, or flatten exposure using a specified order type and limit.
These choices have different costs. A marketable exit can reduce exposure quickly but incur slippage; a passive exit can leave risk open.
Control alert noise
Group repeated alerts by agent, account and failure mode. Debounce recoverable conditions, but do not suppress a new hard-limit breach just because an earlier warning remains open.
Route critical alerts to a monitored on-call channel and include the current exposure, affected orders, last good timestamps and the automated action already taken. An alert that says only API error gives the responder little to do.
Design recovery for the unattended case
Monitoring without an available operator needs bounded recovery behaviour. Retrying forever can worsen an outage; stopping everything can leave working orders exposed. Define a small state machine for degraded operation, such as healthy, degraded, trading-paused, reconciling and recovered. Specify which transitions are automatic and which require approval.
For example, on a market-data gap, stop generating new entries based on the missing feed. On a broker disconnect, stop submissions and retain enough durable state to reconcile after reconnect. Do not assume that cancelling an order succeeded just because the request was sent. On position mismatch, prevent new risk until the system identifies whether the discrepancy came from a missed fill, a manual trade or stale state.
Test recovery by injecting faults, not only by checking that an alert fires. Disconnect the broker after submission but before acknowledgement. Delay fills, duplicate an event, restart the process with a working order, and make the market-data timestamp stop advancing while the socket stays open. Verify that the agent:
- Does not create duplicate orders during retries or restarts;
- Reconstructs open orders and positions from broker state;
- Keeps exits and risk-reduction actions distinct from new entries;
- Records the reason for pausing and the evidence required to resume; and
- Does not automatically resume after an unresolved position or order mismatch.
Store enough telemetry to investigate incidents without retaining unnecessary sensitive data. Make logs append-only where practical, synchronise clocks, and test that alerts arrive when the primary monitoring service itself is unavailable. Keep a simple operational runbook with account identifiers, stop procedures, expected state transitions and a route to disable credentials. Review the runbook after every material incident and after changes to order handling.
For platform selection, test whether the API exposes the order, fill and position events your controls depend on, and whether reconnect behaviour is documented. A platform assessment such as FX-Tech's platform criteria can inform that review, but the monitoring design still needs to be validated against the specific account and strategy configuration.
The useful test is not whether a dashboard turns red. It is whether the system detects a meaningful fault, limits additional exposure, preserves enough evidence to diagnose it, and resumes only when state is known. What failure should cause this agent to stop trading before anyone is available to intervene?
Frequently Asked Questions
What should I monitor in an unattended trading agent?
Monitor process and strategy liveness, market-data age, order state transitions, broker acknowledgements, fills, positions and risk-limit utilisation. Track local state against broker state, especially after timeouts, reconnects and restarts.
How often should a trading agent reconcile positions?
Active strategies often reconcile every 1 to 5 seconds, while low-frequency strategies may use a slower interval. Choose a cadence that catches discrepancies before they can create unacceptable exposure, while respecting API limits.
What should an agent do when an order acknowledgement times out?
Treat the order as having an unknown state. Query by client order ID or reconcile open orders before retrying, because the broker may have accepted the original request even if the acknowledgement was lost.
How can I reduce false alerts from a trading bot?
Set thresholds from measured baselines, use persistence for recoverable delays and group repeated alerts by failure mode. Do not debounce hard risk-limit breaches or unresolved order and position mismatches.
