Designing idempotency and retries for live order placement

On Friday afternoon, a developer looks at the execution dashboard after a bad week. It shows three timeouts, two positions larger than expected and an order that may have been accepted twice.

The broker returned no acknowledgement. The service retried anyway.

This is the central problem with automated order placement: a timeout describes what the client observed, not what the broker did. The request may never have arrived, may have been rejected, or may have filled before the response was lost. Safe systems preserve that uncertainty and resolve it before creating more exposure.

When the acknowledgement disappears

Suppose an agent submits a market buy for 100 shares. The broker accepts it, but the network drops the response. The client sees a timeout. If it sends a new order for 100 shares, it could end up long 200. If it assumes the order failed and does nothing, it could remain flat when the strategy expects a position.

There is no client-side technique that can make a remote request and a local database write atomic. The practical goal is narrower: give each trading intent a durable identity, make repeated submissions refer to that same intent, and query the broker until the order state is known or the system stops safely.

Keep three concepts separate:

• Intent: the strategy decision, such as buy 100 shares of a symbol within a defined price limit.
• Order: the broker-side object created to represent that intent. One intent may produce more than one order if the system deliberately cancels and replaces.
• Attempt: a network transmission or status query. Repeating an attempt must not silently create a new intent.

This distinction matters for market orders, limit orders, and cancel-replace flows. A retry policy should not turn a single decision into multiple independently identifiable orders. Nor should it treat every error as proof that nothing happened.

Start by recording the intent before sending it. Persist its parameters, account, strategy version, creation time and unique identifier. If the service restarts after a request, it can recover the same intent rather than rebuilding it from a fresh strategy calculation.

Give every intent a durable identity

Generate a client order ID once, before the first submission, and persist it with the intent. Reuse that ID for every retransmission of the same unchanged order. A UUID or a random 128-bit value is suitable internally; if a broker limits ID length or character set, map it to a compact value while retaining the full internal identifier.

Do not derive the ID only from symbol, side and quantity. Two legitimate strategy decisions can have identical parameters. Use a unique intent ID, or combine a stable strategy instance ID with a monotonic sequence or UUID. Enforce a database uniqueness constraint so two workers cannot submit the same intent concurrently under different IDs.

Broker semantics differ. Some APIs explicitly deduplicate repeated submissions by client ID. Others use the ID for lookup but do not promise that a repeated POST returns the original order. Some restrict uniqueness by account, trading day, symbol or a retention window. Check the documented scope, maximum length, character set and duplicate response behaviour. Treat undocumented behaviour as no deduplication guarantee.

Maintain a local order record with states such as prepared, submitting, acknowledged, unknown, partially filled, filled, cancelled, rejected and expired. Save the request and transition to submitting in a transaction before making the network call. A transactional outbox can then hand the request to a worker without losing it between a database commit and queue publication.

For each state transition, store the broker order ID, client order ID, last response, timestamp and cumulative filled quantity. Make processing idempotent too: repeated fill events must update the position once, not once per delivery. A practical key is the broker execution ID, protected by a uniqueness constraint. If the venue has no stable execution ID, deduplicate cautiously using its documented event fields and retain raw events for review.

Never change quantity, price, side or time-in-force while reusing an ID unless the broker explicitly defines that operation as an amend. A changed order is a new command with its own lifecycle. This boundary prevents an uncertain original request from being overwritten by a second, different intention.

Retry by state, not by exception class

Classify the outcome before deciding what to do. A local validation error or a clear broker rejection is not a transient transport failure.

A read timeout after the request body was sent is ambiguous. A connection failure before any bytes were sent may be safe to retry, but many client libraries cannot reliably tell whether the body reached the remote endpoint.

For an ambiguous submission, query by client order ID first. If the order is found, attach the broker order ID and continue processing its state. If it is not found, allow for delayed visibility, then query again.

Only submit again with the same client ID if the broker documents duplicate protection for that operation. If it does not, stop automated submission and escalate to reconciliation rather than inventing certainty.

Use bounded backoff with jitter for status queries and explicitly retryable requests. For example, a service might query after 250 milliseconds, then 750 milliseconds, then 2 seconds, with random jitter and a total short investigation window of 5 to 10 seconds. These are implementation starting points, not universal venue timings. Respect Retry-After on rate limits, and avoid synchronised retries across a fleet.

Typical API timings and limits vary by broker, asset class, account tier and venue. The figures below are typical market ranges for planning, not audited performance of any named provider or service.

Use different policies for different response classes. The following is a useful starting split:

• Clear validation or risk rejection: do not retry unchanged. Record the reason and let strategy or risk logic decide whether to form a new intent.
• Rate limit: wait for the stated interval, add jitter and preserve the original intent ID.
• Server error or timeout after submission: treat as unknown, query by client ID and retry only under documented duplicate protection.
• Read-only status query failure: retry with bounded backoff, then mark the order unresolved and raise an alert.

Set a deadline as well as an attempt count. A market order whose submission remains unresolved for several seconds may no longer match the strategy's price assumptions.

Continuing to retry because the network has recovered can create a stale trade. The right action may be to suspend new orders for that symbol or account until exposure is reconciled.

Reconcile exposure before acting

Order state and position state are different sources of information. A filled order can be missing from a delayed position snapshot, and a partially filled order can still have live leaves. Track cumulative filled quantity, remaining quantity and average fill price from execution events, then compare those values with broker-reported orders, executions and positions.

Reconciliation should run after restarts, disconnections and unresolved submissions, not just once each morning. Fetch open orders and recent executions with enough overlap to catch delayed events. Deduplicate executions by their stable broker ID. If an order is absent from the open-order list, that does not prove it never existed; it may have filled, been cancelled or fallen outside the query window.

Keep a short-lived unknown state rather than forcing an order into rejected or failed. During that state, apply a conservative exposure rule.

One option is to reserve the full requested quantity against the account's risk limit until the broker confirms a fill, rejection or cancellation. This can temporarily reduce capacity, but it prevents a second order from using risk budget that may already be committed.

Cancel requests have the same ambiguity as new orders. A cancel acknowledgement may be lost after the order fills, or a fill may race with the cancel.

Do not release reserved exposure merely because a cancel was sent. Wait for a terminal order status and reconcile any fills received during the race. A cancel-replace operation should have explicit parent and child IDs so the system can tell which order may still execute.

For unresolved orders, define an operational response in advance. For example, block additional orders in the affected instrument, poll at a slower rate after the initial window, alert an operator and keep the account in a restricted state until broker data agrees with local records.

A paper-trading environment can exercise these paths without risking capital. Agentic Trading Index offers agentic paper trading from $9.99 per month, and a paper trading environment for trading agents can help test duplicate submissions, delayed acknowledgements and partial fills before live deployment.

Test the failure path, not just the happy path

A sandbox that returns a clean acknowledgement for every request will not prove retry safety. Inject failures at specific boundaries: after the broker receives the request but before the response reaches the client; after the database records submission but before the worker sends it; and while an execution event is delivered twice. Confirm that each case leaves one intent, one correctly tracked order and one position update per actual fill.

Test concurrent workers as well. Two processes can claim the same queued intent if a lease expires during a slow network call.

Use database constraints, atomic claims or a lease with a fencing token. A worker that has lost its lease should not be allowed to submit a new order merely because it eventually receives a response.

Measure more than retry count. Track the share of submissions ending in unknown state, time to resolve, duplicate client-ID responses, duplicate execution events, reconciliation differences and reserved exposure held while status is unclear. Alert on changes from your own baseline. A rising rate of unknown orders can signal a broker outage or a client timeout configured below normal latency.

Before live use, verify that the system can recover after a process restart, respect rate limits, preserve IDs across deployments and stop when broker state cannot be established. Keep the original request payload and each response for diagnosis, while protecting credentials and sensitive account data. A manual review path is a control, not a substitute for deterministic order state.

The mistake most readers will still make is treating a timeout as a rejection and sending a fresh order with a fresh ID.

#order reconciliation #automated order placement #duplicate orders #client order ID #trading API errors #order retry logic #broker API #execution system