"How will we know the agent is safe if nobody is watching it?" Build a separate monitor that checks the agent, market data and broker state, then takes a defined action when any of them stops matching expectations. A log file and a green process heartbeat are not enough. A process can be alive while sending duplicate orders, trading on stale prices or holding a position the strategy does not know about.
Monitoring is not a substitute for risk limits. It is the mechanism that detects when those limits, inputs or assumptions are no longer holding. Make every alert measurable, assign it a response and test that response before the system is left unattended.
Monitor the agent from outside its process
If the agent reports its own health, a crash or deadlock can silence the report as well. Run a watchdog in a separate process or service. It should receive heartbeats, inspect persisted state and query the broker independently. Keep the watchdog's credentials and deployment boundary separate enough that a failure in the strategy process does not disable the monitor.
Emit structured events for each decision and state transition. At minimum, record a timestamp, strategy and account identifiers, instrument, client order ID, broker order ID, event type, quantity, price, position and reason code.
Use a monotonic clock for durations and synchronise system clocks against a reliable time source. Include a correlation ID from signal through submission, acknowledgement, fill and position update. Without it, a burst of retries is difficult to distinguish from several intended orders.
Track the age of the last valid market-data update, the last broker response, the most recent position reconciliation and the last successful risk check. Measure order acknowledgement latency and the time an order has remained in each state. A heartbeat says only that code executed recently. It does not prove that the code received fresh data or made progress towards a known state.
Keep monitoring independent of the trading loop. A metrics backend, alert router and durable event store can fail too, so expose their health and provide a fallback path such as a second notification channel. If an alert only appears on a dashboard that nobody is viewing, it is not an operational control.
Set thresholds that lead to a decision
Start with a baseline from the actual instruments, venues and order types the agent uses. The ranges below are typical starting points, not universal limits or audited results.
A quiet market-data stream can be normal for one instrument and a feed failure for another. Set thresholds against the feed's expected cadence and the strategy's maximum tolerable delay.
| Signal | Typical ranges | Example response |
|---|---|---|
| Agent heartbeat | Every 1-5 seconds; investigate after 2-3 missed intervals | Stop new submissions if the watchdog cannot confirm progress within the configured grace period. |
| Market-data age | About 1-5 seconds for many liquid intraday feeds; set per instrument and feed | Mark the instrument unavailable and block decisions based on stale quotes. |
| Order acknowledgement | Often 100 milliseconds to 2 seconds; alert after 5-10 seconds if that exceeds the venue baseline | Query by client order ID before retrying. Do not assume the first request failed. |
| Position reconciliation | Every 5-30 seconds for active strategies; 1-5 minutes for lower-frequency use | Block additional exposure when the local and broker positions differ beyond tolerance. |
| Reject rate | Often below 1-5% in stable operation, depending on order type and venue | Alert on a sustained breach of the strategy baseline, such as a 2-3x increase over a rolling window. |
| Risk-limit utilisation | Warn near 70-85%; define a hard stop before 100% | Reduce or stop new risk according to the account's explicit limit policy. |
Ranges are useful for triage, but avoid treating a single generic percentage as safe across all strategies. A maker strategy may have routine cancellations and rejects that would be abnormal for a slower portfolio rebalance. Track both absolute thresholds and deviation from a rolling baseline. Use windows long enough to avoid paging on one transient response, but short enough to stop a damaging burst. For many active systems, windows of 1-5 minutes are a practical starting point for rates; fast exposure checks may need to run on every order.
Define severity by consequence. A delayed informational metric can be logged.
A stale quote, unknown order state or position mismatch can require the agent to stop increasing exposure. Route urgent alerts to a channel that creates an acknowledged incident, not just an email inbox. The response path should work overnight, through weekends and during the venue's most active session.
Reconcile broker state, not just local events
The broker is the source of truth for accepted orders, fills and account positions. Local event processing can drop a message, process it twice or restart between an order submission and its acknowledgement.
Poll or subscribe to broker state and compare it with the agent's durable order ledger. Reconcile open orders, fills, cancellations, positions, cash and any relevant margin figures.
Give every intended order a durable client ID before submission. On a timeout, first query the broker for that ID.
If the request may have reached the venue, resubmitting blindly can create a second live order. Treat order states explicitly: pending submission, acknowledged, partially filled, filled, cancel pending, cancelled, rejected and unknown. Unknown is a real state, not a reason to guess.
Make reconciliation differences actionable. A position mismatch of one share can be material for a small account and irrelevant for a large one.
Set tolerances by instrument and account, and specify how long a mismatch can persist before blocking new orders. Record the compared values and timestamps so an operator can tell whether the discrepancy came from a delayed fill event, a manual trade or a genuine ledger error.
A useful monitor checks after reconnects, process restarts and market-open events, as well as on a regular schedule. It should also check that open orders still exist and that cancelled orders are no longer working. For an unfamiliar broker or strategy, rehearsing these state transitions in an agentic paper-trading environment can expose gaps before live access. Paper trading from fx-tech.ai starts at $9.99 a month. Treat the test as a rehearsal, not proof that live venue behaviour will match it.
Choose safe actions for each failure
An alert without a response plan leaves the system in the same state that triggered the alert. Define actions in advance, and make the first automated action conservative. Depending on the fault, that may mean blocking new orders, cancelling working orders, reducing order size or switching to a known safe mode.
These options have different risks. Cancelling every order during a disconnection may fail, while flattening a position with stale prices can make the situation worse.
Write a short runbook for each high-severity condition. It should say what the monitor observed, what action it took, what it could not confirm and what an operator must check before resuming.
A safe restart requires fresh market data, a successful broker reconciliation and a clean risk check. Do not let a process restart clear a latched stop condition.
- Stale data: block new decisions for affected instruments. Resume only after valid updates arrive and pass sanity checks.
- Unknown order state: stop submissions for that order chain, query the broker and reconcile before attempting another order.
- Position mismatch: halt additional exposure in the affected account. Preserve the discrepancy and its timestamps for investigation.
- Risk limit breach: apply the account's pre-agreed response. Do not assume that cancelling orders or flattening is always the correct action.
Test both the trigger and the action. Inject a missed heartbeat, delayed acknowledgement, duplicate fill event, dropped connection and position mismatch.
Confirm that alerts arrive, the state is persisted and the agent cannot resume by itself when the underlying issue remains. Record recovery time as well as detection time. A ten-second detection followed by an hour of uncertain manual recovery is still a serious operational gap.
Review monitoring as an operating control
Give the team one view that answers four questions: Is the agent progressing? Are its inputs fresh? Does the broker agree with its ledger? Is risk inside the configured limits?
Show current values beside thresholds and include the age of each observation. A dashboard full of charts is less useful than a small set of readable states with links to the relevant order and event history.
Keep a daily report of order counts, fill and reject rates, acknowledgement latency, cancellations, position differences, stale-data incidents and time spent in safe mode. Compare the figures by strategy, instrument and venue. If alerts are firing repeatedly without a useful action, revise the threshold or improve the response.
Do not simply mute them. If an incident occurs, preserve event logs and configuration versions so the observed behaviour can be reproduced.
Before unattended operation, use a staged release. Run the monitor against replayed events, then paper orders, then low-risk live orders with explicit exposure caps. Increase limits only after the reconciliation and recovery paths have behaved as expected. Paper environments can help rehearse alerts and state handling; for example, FX-Tech's paper-trading tools provide a place to exercise agent workflows before live deployment. Keep venue-specific live checks in the release plan because simulated acknowledgements, liquidity and failure modes may differ.
Set an owner for each alert, a review interval for each threshold and a versioned record of the policy. Review after broker API changes, strategy releases, instrument additions and incidents. A threshold that was reasonable for a small pilot may be too loose after order volume or account exposure grows. Go and check the age, in seconds, of the oldest unresolved order-state mismatch in the current deployment. It should read 0 before the next unattended session starts.
Frequently Asked Questions
What should a trading agent monitor while running unattended?
Monitor heartbeat and decision progress, market-data age, broker connectivity, order-state transitions, fills, positions and risk-limit utilisation. Compare local records with broker state, since a healthy process can still have an incorrect view of its orders or exposure.
How often should an automated trading system reconcile positions?
Active strategies often start with checks every 5-30 seconds, while lower-frequency systems may use intervals of 1-5 minutes. Reconcile after reconnects, restarts and material order events as well, and tune the cadence to the cost of a missed or delayed fill.
What should happen when an order acknowledgement times out?
Treat the order as unknown, then query the broker using its durable client order ID before retrying. Blind resubmission can create duplicate exposure if the original request was accepted but its acknowledgement was lost.
Should a trading agent cancel orders automatically when an alert fires?
Only if the alert's response policy says cancellation is appropriate and the system can verify the result. During a connection failure, cancellation may not reach the venue, and flattening a position on stale prices can add risk.
How can I test monitoring before live deployment?
Replay delayed and duplicated events, disconnect the broker interface, and inject stale data, missed heartbeats and position mismatches. Check that alerts arrive, state persists and the agent stays blocked until reconciliation and risk checks pass.
