A timed-out order request is not evidence that the broker rejected it. The broker may have accepted the order while the response disappeared in transit. Retrying blindly can double the position; refusing to retry can leave a strategy underfilled. The useful question is not how many retries ran, but whether each retry was safe and whether the resulting exposure matched the intended order.
The unit of idempotency is an order intent
Build a stable intent fingerprint
Give every intended order a durable identifier before sending it. Persist that identifier with the account, strategy, instrument, side, order type, quantity, limit price and time-in-force. A retry of the same intent must reuse the identifier and the same material order parameters. A changed quantity or price is a new intent, not another attempt at the old one.
Where the API supports idempotency keys, confirm their scope and retention. Some apply only to a particular endpoint or account, and some expire. Others reject a repeated key if the payload differs. If the broker offers a client order ID instead, generate it locally and store the mapping before submission. Never assume that a request ID generated after a response arrives can protect a request whose response was lost.
Keep a local state machine rather than a single sent flag. Useful states include prepared, submitted, acknowledged, partially filled, filled, cancelled, rejected and outcome unknown.
Store broker order IDs, execution IDs, timestamps and the latest cumulative filled quantity. This makes the order intent the stable record, while individual HTTP or socket submissions become attempts attached to it.
Treat timeouts as unknown outcomes
On a timeout after sending, mark the intent unknown. First query the broker by client order ID or inspect order and execution updates. Only replay when the venue's documented semantics make that replay safe, or when reconciliation establishes that the original request was not accepted. A cancel-replace is a separate operation and needs its own state handling: the original may fill while cancellation is pending.
Retries also need classification. Connection failure before any bytes leave the process can be different from a timeout after the full request was written.
A validation error, authentication failure or insufficient-margin response normally needs correction, not repetition. A rate limit should follow the broker's retry guidance. A 5xx response or lost connection after submission can leave acceptance uncertain, so check the order state before sending again.
- Reuse one intent ID for a true replay, and reject a payload mismatch locally.
- Record each attempt separately, including send time, response, timeout reason and backoff.
- Never infer "not accepted" from the absence of an acknowledgement alone.
Measure retries as a lifecycle, not a counter
Choose denominators that expose failure
A raw retry count rewards systems that retry aggressively and hides whether they create extra orders. Track attempts per distinct intent, but pair it with outcome measures.
Define a duplicate accepted-order rate as accepted broker orders beyond the intended order count, divided by submitted intents. Track duplicate fills separately: multiple accepted orders may be cancelled before they trade, while one duplicate fill can create real exposure.
Measure retry recovery as timed-out intents later matched to an accepted broker order or a terminal rejection, divided by all timed-out intents. Segment the result by venue, endpoint, instrument, session and failure class. A low recovery rate could indicate missing broker history, weak client-ID lookup or genuinely lost submissions. It is not automatically a reason to increase retries.
Track ambiguity duration from the first uncertain submission to a reconciled terminal state. Report median, p95 and p99, not just the mean. A five-second median can coexist with a several-minute tail during a reconnect or broker maintenance window. During that tail, the strategy should reserve the maximum plausible exposure and avoid submitting a replacement that assumes the first order is absent.
The ranges below are typical starting ranges for liquid instruments on broker APIs, not guarantees. Validate them against the venue, asset class, connection method and session conditions before using them as alert thresholds.
| Measure | Typical range | How to use it |
|---|---|---|
| New-order acknowledgement, p95 | 0.2-2 seconds | Set a timeout from observed venue latency, not a fixed universal value. |
| Initial retry or reconciliation delay | 0.25-1 second | Allow a short response window, then query order state before replay. |
| Exponential backoff cap | 2-10 seconds | Reduce request bursts during a degraded API period. |
| Unknown-outcome escalation | 5-30 seconds | Pause new risk in the affected instrument or strategy while resolving state. |
Separate latency from safety
Fast acknowledgements do not prove safe retries. Track submission-to-acknowledgement latency, but also measure intent-to-reconciled-state latency and the share of uncertain requests that resolve without a second accepted order. Set distinct service objectives for operational speed and exposure safety. For example, a team might alert when p99 ambiguity exceeds its 10-second operating budget, while requiring zero known duplicate fills in its test runs. Those are locally chosen objectives, not universal market standards.
Calculate retry amplification as outbound submissions divided by distinct order intents. A ratio of 1.03 may be ordinary in a system with occasional transient faults; a sudden increase to 1.4 can signal a reconnect loop, a short timeout or a broker response change. Interpret it beside accepted-order count, reject rate, fill rate and API error class. A higher attempt rate with unchanged recovery and rising ambiguity is a warning, not resilience.
Instrument the decision to replay
For each attempt, emit structured events with a shared intent ID and a unique attempt number. Include account, strategy, instrument, venue, payload hash, client order ID, request start and finish times, transport outcome, broker response code, broker order ID, cumulative filled quantity and reconciliation result. Keep clocks synchronised and retain both local and broker timestamps. Without those fields, an apparent duplicate may be a partial fill, a legitimate child order or a delayed event.
Make retry policy observable as data. Log the decision reason, the evidence consulted and the delay selected. This lets a review distinguish "replayed because the connection failed before write" from "replayed because no order appeared in a query after a timeout". Keep secrets and personal data out of logs, but retain enough identifiers to join requests, order updates and executions.
Dashboards should group events by intent and show an order timeline, rather than presenting disconnected request counts. Alert on measures that imply unmanaged exposure: duplicate accepted orders, duplicate fills, unresolved intents past the risk budget, and repeated attempts without a state change. A retry-rate alert alone can page on harmless read-side failures while missing the dangerous case of one ambiguous market order.
Use a controlled agentic trading test environment to exercise delayed acknowledgements, dropped responses and reconnects before live deployment. The test should verify both broker-side order state and the agent's local exposure estimate. A response saying "accepted" is not enough if the agent still submits again after losing the response.
Test recovery under deliberate faults
Build a fault matrix around the point where uncertainty enters. Include a connection drop before request write, a drop after write but before response, a delayed acknowledgement, a duplicate event, an out-of-order fill update, a 429 response and a restart while an order is unknown. For each case, specify expected attempts, resulting broker orders, local position and escalation behaviour. Run the matrix for market, limit and cancel-replace flows where those order types are supported.
- Submit a unique intent and persist it before network transmission.
- Inject a failure at a chosen point, then restart the process if the scenario requires it.
- Allow the normal reconciliation path to run, without manually repairing local state.
- Compare broker orders and executions with the agent's intent ledger and calculated position.
Measure the pass rate by scenario, not only as one aggregate number. Also record recovery time, excess accepted orders, excess filled quantity and the number of intents left unresolved at the end of the test window.
A clean average can hide a single failure mode that adds a full second position. Keep a small, reviewed set of fault cases in continuous integration, then run longer venue-specific tests around reconnects and session boundaries.
Retries improve availability only when the system can distinguish a lost response from a lost order. Which unresolved order state would force the strategy to stop adding risk in the next team meeting?
Frequently Asked Questions
What does idempotency mean in automated order placement?
It means repeating a request for the same order intent does not create additional orders. The client and broker need a stable identifier, consistent payload rules and documented key scope for that guarantee to hold.
Should an order request be retried after a timeout?
Not until the system checks whether the broker accepted it. Query by client order ID or reconcile order and execution updates first; otherwise a replay can create duplicate exposure.
Which retry metrics should a trading system track?
Track attempts per intent, duplicate accepted orders, duplicate fills, retry recovery rate and time spent in an unknown state. Segment them by venue and failure type so aggregate figures do not conceal a specific unsafe path.
How should a system handle brokers without idempotency keys?
Generate and persist a unique client order ID before submission, then use it to query the broker after ambiguous outcomes. Confirm that the broker supports reliable lookup by that ID and define a safe escalation path when order state remains uncertain.
