The timeout that creates a second order
A single duplicated 100-share order at $50 ties up $5,000 of unintended exposure before fees. If the first request reached the broker but its response did not reach your service, a blind retry can place that second order while the first is already live.
That is the core problem order idempotency addresses. A client sends an order request, the network or broker times out, and the client cannot tell whether the broker accepted it. The timeout describes what the client observed, not what happened at the other end.
Retries are still necessary. Networks fail, gateways return temporary errors, and services restart.
The aim is not to avoid retrying. It is to make repeating an instruction safe, or to pause and discover the actual order state before sending it again.
Start with three outcomes for every submission: accepted, rejected, or unknown. A clear rejection can often be handled according to its reason.
An accepted order should be tracked by its broker order ID. An unknown outcome needs reconciliation, not a fresh order with a new identifier.
A timeout is an unknown outcome, not proof that an order failed.
Exactly-once delivery across a client, network and broker is generally not something an application can guarantee on its own. A solid design combines stable client intent, broker-side duplicate protection where available, bounded retries and checks against broker-reported orders and fills.
Give each trading intent a stable identity
Create an internal intent record before making the network call. Give it a unique identifier, such as a UUID, and persist the intended account, instrument, side, quantity, order type, limit price and strategy decision. The same economic instruction must keep the same identity through process restarts and retries.
If the broker supports a client-supplied idempotency key, pass that stable identifier on each retry. The broker may then return the original result instead of creating another order. Read the API rules carefully: key scope may be per account or endpoint, retention may be limited, and a repeated key with a changed payload may be rejected or handled differently.
A client order ID is useful for finding and reconciling an order, but it is not automatically an idempotency guarantee. Some APIs accept the same client ID more than once, or enforce uniqueness only for a limited period. Test the behaviour and document the broker's exact contract before relying on it.
Persist the intent before sending it. A practical record includes the key, a canonical payload hash, creation time, attempt count, last response, broker order ID if known, and current state.
The hash helps detect an unsafe case: the service is reusing an identifier while changing quantity or price. Treat that as a new intent, or use the broker's explicit amend or cancel-and-replace flow.
For services with a database and a message queue, a transactional outbox can reduce another failure window. Commit the intent and an instruction to submit it in one database transaction, then let a worker deliver it. The worker may still send twice if it crashes after transmission, so stable identity and reconciliation remain necessary.
Keep order identity separate from execution identity. One accepted order can produce several partial fills, each with its own execution ID. Deduplicate fill events using the broker's execution identifier, not the order ID, or later partial fills may be discarded as repeats.
Retry only when the failure is understood
Classify errors before choosing an action. A validation error such as an invalid tick size is usually permanent until the payload is fixed. A rate limit may be temporary and include a retry-after value. A connection reset or response timeout after sending is ambiguous: the broker may already have accepted the order.
The figures below are typical market ranges for planning and testing, not guarantees for a particular broker. Measure the actual API under the account, venue and order types the system will use.
| Situation | Typical market range | Operator response | Duplicate risk |
|---|---|---|---|
| Order submission response | About 100 ms to 2 s in normal conditions | Set a deadline from measured latency, often 2 to 5 s | High if a timeout triggers a new order |
| Temporary rate limit | Retry-after often 1 to 10 s, where supplied | Respect the broker value and reduce request rate | Low if the request outcome is known |
| Gateway or service error | Recovery can take seconds to minutes | Use bounded backoff, then reconcile | Medium to high if acceptance is ambiguous |
| Idempotency key retention | From minutes to several days, depending on API | Check the documented scope and expiry | High if a late retry falls outside the window |
For retryable errors with a known non-acceptance outcome, exponential backoff with jitter prevents many clients from retrying in lockstep. A typical policy might begin around 250 to 500 ms, double the delay, add random jitter, and cap at several seconds.
Set a maximum attempt count or total retry window. A strategy should not keep trying indefinitely while market conditions change.
For an ambiguous timeout, first query open orders or search by the client order ID. If the broker confirms the order, attach its broker ID to the existing intent and continue monitoring it.
If the lookup is temporarily unavailable, preserve the unknown state and retry the lookup. Do not submit another order merely because the first lookup returned no result; the broker's search may be delayed or incomplete.
Use this submission sequence as a starting point:
- Persist the intent, payload hash and stable client key.
- Submit once, recording the request time and response or transport error.
- On a clear acceptance, store the broker order ID and monitor order updates.
- On a clear rejection, apply a reason-specific correction or stop.
- On an unknown outcome, query by client key and reconcile before any resubmission.
Reconcile the broker state, not your assumption
Your local database is a record of what the application attempted. The broker's order and execution reports describe what the market-facing system accepted and filled.
Neither view is sufficient by itself during a failure. Reconciliation compares them and resolves differences without inventing a second trade.
On startup, after a stream disconnect, and after any ambiguous submission, fetch open orders and recent executions for the relevant account. Match using broker order ID first and client order ID where available. Include terminal states such as rejected, cancelled, expired and filled. A partially filled order remains an order with a remaining quantity, not permission to resend its original full quantity.
Order updates can arrive late or out of sequence. Store event timestamps and broker sequence numbers when available, but do not rely on arrival order alone. Treat cumulative filled quantity as a monotonic value when the API defines it that way, and deduplicate individual executions. If a correction or bust is possible for the instrument, represent it as a separate adjustment rather than silently rewriting history.
Reconciliation also has a cost. Frequent polling consumes rate limits and can create load during an incident. Use streaming updates for normal operation, then poll after a disconnect or unresolved state.
A typical polling interval might be 1 to 5 seconds during active recovery and 15 to 60 seconds for slower checks, subject to broker limits and the risk of the strategy. These are starting ranges, not universal settings.
Define a safe response when state cannot be resolved. For a small passive order, the system may hold new submissions in that symbol while it continues looking up the original.
For a strategy with rapidly changing exposure, it may need to stop all new orders for the account and alert an operator. The choice depends on maximum position, order urgency and how much duplicate exposure the risk budget permits.
- Track submitted, acknowledged, partially filled, filled, cancelled, rejected and unknown as distinct states.
- Compare order quantity, remaining quantity and cumulative fills before creating a replacement.
- Record every reconciliation decision, including the data source and timestamp.
- Escalate if local and broker state still disagree after a defined recovery window.
These controls can be exercised in a paper environment before live deployment. An agentic trading paper environment such as FX-Tech can help test order workflows, but confirm which broker-specific duplicate, latency and recovery behaviours it models.
Test the failure path before unattended operation
A happy-path test proves that one request can create an order. It says little about retries. Build fault injection into the execution service: drop the response after the broker accepts, delay it past the client deadline, return a rate limit, restart the worker between send and persistence, and deliver duplicate or out-of-order updates.
For each test, assert the resulting position and order count, not just the HTTP response. A useful invariant is that one intent can create no more than one live broker order, unless the system records an explicit cancel-and-replace transition. Another is that the sum of unique executions equals the broker-reported filled quantity, allowing for documented corrections.
Test key reuse after a process restart and after the broker's idempotency window expires. Verify how the API responds to the same key with an identical payload, then with a changed payload.
Test what the order search returns when the broker accepted an order but its search index has not caught up. Those cases expose assumptions that a sandbox may hide.
Set operational limits as well as code-level rules. Cap retries per intent, total requests per account, and unresolved unknown orders.
Emit metrics for timeouts, duplicate-key responses, unknown-state duration, reconciliation mismatches and retries by error class. Alert on sustained changes, not only on a single failed request.
Keep a manual intervention path, but do not make an operator the normal deduplication mechanism. Log enough information to identify the intent and compare it with the broker without exposing credentials. Document how to pause new orders, resolve an unknown submission and resume without replaying the original instruction.
If this were my operation, I would first add a durable intent record and stable client order key, then test the case where acceptance succeeds but the response is lost. I would not permit unattended retries until reconciliation can prove that the same intent cannot create a second live order.
Frequently Asked Questions
What does idempotency mean in order placement?
It means repeating the same order instruction does not create another order. In practice, a stable key and broker-side support can provide this behaviour, but the API's key scope and retention rules must be checked.
Should an order be retried after a timeout?
Not immediately as a new order. The broker may have accepted the original request, so query by its client order ID or reconcile open orders and executions first. Retry with the same key only if the broker documents safe idempotent handling.
Is a client order ID the same as an idempotency key?
Not necessarily. A client order ID helps identify an order, while an idempotency key may prevent repeated submissions from creating duplicates. Some APIs use one value for both purposes, but their behaviour and expiry need testing.
How should an automated trading system handle HTTP 429 errors?
Respect a broker-provided retry-after value and reduce request frequency. Keep retries bounded, add jitter if the delay is not specified, and distinguish a confirmed rate-limit rejection from an ambiguous timeout after submission.
How can duplicate orders be tested before live trading?
Simulate a broker accepting an order while the client loses the response, then restart the worker and trigger its retry path. Assert that only one live order exists and that broker fills reconcile to the local position.
