Idempotency and retries: where order systems go wrong

One operator sees a submit request time out and retries it immediately. Another records the order as uncertain, checks broker state, then retries only if the original request is absent.

The first may get a fill and a duplicate. The second may wait a few seconds longer, but keeps the intended exposure bounded.

That difference is the practical meaning of idempotency in an unattended trading system. A network timeout says the client did not receive a response. It does not say whether the broker accepted, rejected or filled the order. Treating those as the same event is the failure behind many duplicate-order incidents.

Retries do not make an order safe

Retries are useful because connections fail, gateways return errors and responses get lost. They are dangerous because order submission is a side effect. A retry can repeat that side effect even when the first request succeeded remotely.

Suppose a strategy wants to buy 100 shares. The client sends the order, the broker accepts it, and the acknowledgement disappears during a connection reset. If the client sends a fresh order for 100 shares, it has not retried the same operation. It has asked for another 100 shares. The account can end up long 200.

There is no general exactly-once delivery guarantee across a client, network and broker. A solid design combines a durable record of intent, a stable identifier, broker-side duplicate protection where available, and reconciliation against broker state.

Even that depends on the broker's documented rules. Some APIs accept a client order ID as a deduplication key; others use it only for lookup, or enforce uniqueness only while an order is open.

Make the uncertain outcome explicit in the application state. A useful order lifecycle includes prepared, submitted, acknowledged, partially filled, filled, cancelled, rejected and unknown.

Unknown is not a synonym for failed. It is a state that blocks an unsafe new submission until the system has evidence to move on.

Give each trading intent one durable identity

Create an intent record before sending anything to the broker. It should capture the account, instrument, side, quantity, order type, limit or stop price, time in force, strategy decision and a unique client order ID.

Commit that record to durable storage first. A process restart must not turn one strategy decision into a new order identity.

A deterministic intent key can prevent duplicate work upstream. For example, combine a strategy instance, decision sequence and account identifier, then store the resulting client order ID alongside the immutable order payload.

Avoid generating a new random ID every time a queue worker handles a message. Most job queues deliver at least once, so the same message may reach a worker repeatedly.

• Enforce a database uniqueness constraint on the intent key and on the client order ID within the broker account.
• Persist the exact payload sent, including quantity, price, order type and time in force. A retry with a changed payload is a new decision, not a replay.
• Record every attempt and response separately from the intent. Keep timestamps, request IDs, transport errors and broker order IDs.
• Check the broker's ID length, allowed characters, uniqueness scope and retention rules. Do not assume an identifier remains searchable forever.

Broker-side idempotency can help, but only if its contract is understood. Confirm whether the key applies to POST requests, how long it is retained, what happens when the same key arrives with a different payload, and whether a duplicate returns the original order or a conflict. If those rules are missing, treat the feature as unavailable until tested and confirmed.

An order replacement needs its own semantics. A cancel-and-replace workflow can race with a fill: the old order may execute while the cancel is in flight, then the replacement may also be accepted. Link the replacement to the original intent, track both broker order IDs, and recalculate the remaining quantity from confirmed fills before submitting anything new.

Choose retry rules by outcome, not by exception class

A broad catch-and-retry block is a common source of duplicate exposure. Classify the outcome first.

A local validation error or an explicit broker rejection is different from a read timeout after the request body was sent. A 429 response may be retryable after its specified delay; a 400 response usually requires a corrected order. A 5xx response can still leave acceptance uncertain if the broker performed the action before failing to return a response.

Use bounded exponential backoff with jitter for requests that are safe to repeat. Honour rate-limit headers and broker guidance.

For an uncertain submission, do not blindly replay with a fresh identity. Query by client order ID or broker order ID, and only resubmit the same intent when the broker's lookup path gives a reliable negative result. If lookup is eventually consistent, allow for that delay before treating absence as proof.

The figures below are typical engineering planning ranges, not measured results for any particular broker. Broker contracts, venue load and instrument liquidity vary. Treat them as starting points for tests and tune them against observed response times and documented limits.

Separate the retry budget for reads from the policy for writes. A quote request can often be repeated without changing exposure. A submit request needs a state check.

A cancel request also needs a state check because the order may have filled before the cancellation took effect. Keep retries bounded by time and attempt count, and surface unresolved orders to an operational alert rather than retrying indefinitely.

Reconcile broker state before changing exposure

After a submission timeout, freeze the intent in unknown state. Query order status using the client ID if supported.

If the broker has no reliable lookup by client ID, use the available order list and match cautiously on account, instrument, side, quantity, price and time window. Matching on symbol and side alone is not enough when a strategy can submit several similar orders.

Process broker events as updates to an order, not as commands to submit another one. Acknowledgements can arrive late or out of order. Store the broker's cumulative filled quantity and execution identifiers. Deduplicate fills by execution ID, then derive remaining quantity as original quantity minus confirmed fills. Do not add a cumulative filled amount to a previous cumulative amount; doing so counts the same execution twice.

• If the order is found open, attach its broker ID and continue tracking it. Do not submit a second order.
• If it is partially filled, update the position and remaining quantity before deciding whether to cancel or replace.
• If it is filled, close the intent and reconcile the resulting position against the account.
• If it is rejected, record the broker reason. A corrected order should be a new, linked intent.
• If no order is found, repeat the lookup within a bounded consistency window. Escalate or halt the affected strategy when absence remains uncertain.

Keep risk controls active during uncertainty. Reserve the full intended exposure while an order may still be live, or use a conservative estimate based on confirmed fills and the maximum remaining executable quantity. Releasing the reservation just because the client timed out lets another strategy use risk capacity that may already be committed at the broker.

Reconciliation should compare more than order status. Compare open orders, executions, positions and buying power.

A broker may report a fill before the local event consumer processes it, or the local process may miss a cancel acknowledgement. A scheduled account-level reconciliation can catch these gaps, but it should not replace the immediate check after an uncertain write.

Test the failure path, not only the happy path

A sandbox test that submits an order and receives an acknowledgement verifies little about retry safety. Inject a dropped response after the broker has accepted the order. Restart the worker between persistence and submission, and again between submission and response handling. Deliver the same queue message twice.

Delay status events, reorder them, and repeat fill messages. The expected result is one intent and no unintended increase in exposure.

Test the broker's exact client ID behaviour with the same key and identical payload, then with the same key and a different payload. Test how quickly a new order appears in lookup results after acceptance, and whether a cancelled order can still report a late fill.

Where a sandbox does not reproduce these behaviours, record the gap and use conservative live controls rather than assuming production is identical. A paper environment such as agentic paper trading can help exercise the client-side state machine before live orders, but it cannot establish a broker's production idempotency contract.

Track operational measures that expose weak recovery logic: unknown-order count, time spent in unknown state, duplicate-intent rejections, reconciliation mismatches, and the age of the oldest unresolved order. Set thresholds by the strategy's market and risk limits. For liquid intraday orders, a few seconds of uncertainty may be material; for a resting limit order, the system may need to maintain state for hours.

Most retry designs fail because they optimise for a fast response instead of a correct account state. Durable identity prevents one decision from becoming several orders.

Reconciliation resolves uncertainty. Explicit risk reservations contain the damage while the answer is still unknown. Which failure can currently cause this system to submit the same trading intent twice, and what evidence would stop it?

#order reconciliation #partial fills #automated order placement #broker API retries #duplicate orders #client order ID #trading system reliability #execution state machine