Devin Hartcastle
What happens if the order API times out after accepting my order? Treat the outcome as unknown, not failed: persist the order identity, check broker state, and retry only when evidence shows the first request did not create an order. This 30-day plan turns that rule into code and operating controls.
Days 1 to 7: define what counts as the same order
Map the broker's actual behaviour
Start by documenting what the API guarantees, rather than assuming a timeout means rejection. Check whether it accepts a client-supplied order ID, whether that ID is unique per account or per day, how long the broker retains it, and what happens when the same ID arrives twice. Verify whether order lookup can find an order by that ID, and whether a cancel-and-replace request has separate identity rules.
Run controlled tests in a paper or sandbox account. Submit a valid order, then submit it again with the same ID. Repeat after a simulated client timeout, with a changed price, and after a partial fill. Record the response codes and the resulting order state. Some APIs treat a repeated ID as a lookup, some reject it, and some offer no deduplication guarantee. Design for the behaviour you observe, then confirm it against the broker's documentation.
Give each trading intent a durable identity
Create the identity before sending any network request. A useful key combines the strategy instance, source signal or event ID, instrument, side, and leg number. Persist it with the intended quantity, order type, limit price, account, creation time and strategy version. Do not derive the key from a timestamp generated anew on every retry.
Keep the intent separate from the broker order ID. The intent is your system's stable record of what it meant to do; the broker ID identifies an order the venue accepted.
If a trader or strategy deliberately changes the price or quantity, create a new revision or replacement intent and link it to the original. Do not silently reuse an ID for a materially different instruction.
- Write the intent and an outbound submission record to durable storage before sending.
- Use a unique database constraint on the intent key to stop duplicate workers creating separate submissions.
- Record every request attempt, response, timeout and subsequent broker observation against that intent.
This gives you a local audit trail even when the response is missing. It does not make a broker without deduplication support exactly-once. In that case, the safe approach is to pause an ambiguous submission and reconcile it before sending again.
Days 8 to 14: implement bounded submission and retry rules
Separate transient errors from unknown outcomes
Classify responses before writing retry logic. A validation error or other clear client rejection needs a corrected instruction, not another identical request. A rate limit may be retried after the broker's stated delay.
A connection failure before any request bytes were sent is usually safe to retry. A read timeout after sending the request is ambiguous: the broker may have accepted it.
Use the same idempotency key for every attempt to submit the same intent. If the broker does not support that key, do not automatically resend after an ambiguous timeout.
First query by client ID, broker order ID if known, and recent account activity. If lookup is eventually consistent, wait and check again. A short delay followed by repeated lookup is safer than treating an empty first response as proof that no order exists.
Keep retries bounded. Use exponential backoff with jitter for transient failures, respect any Retry-After value, and set a maximum attempt count and elapsed time.
The limits should fit the strategy's execution horizon. A marketable order with a short signal lifetime may need an immediate unresolved-state alert; a passive order may allow a longer recovery window. Do not let a retry loop turn a stale signal into a late trade.
The figures below are typical market and implementation ranges, not broker guarantees. Confirm limits, timeouts and ID retention with each API you use.
| Condition or control | Typical range | Action |
|---|---|---|
| Submit request timeout | 2 to 10 seconds | Mark outcome unknown if request may have reached the broker. |
| First retry delay for a transient error | 250 milliseconds to 2 seconds | Apply jitter and increase delay on later attempts. |
| Backoff cap | 10 to 60 seconds | Stop at the strategy's deadline and escalate unresolved orders. |
| Active order status polling | 1 to 5 seconds initially, then 10 to 30 seconds | Reduce polling after the order becomes stable. |
| Client order ID retention | 24 hours to 30 days | Confirm the broker's scope and retention before relying on deduplication. |
Days 15 to 21: reconcile state and protect exposure
Build recovery around order state, not request state
Model order lifecycle states explicitly: created, submitting, unknown, accepted, partially filled, filled, cancelled, rejected and expired. Store broker observations with timestamps. On restart, load unresolved intents and reconcile them against open orders, recent executions and order history before accepting new work. If the broker's query endpoints disagree, keep the intent unresolved and avoid submitting a substitute order automatically.
Partial fills need special handling. If an order for 100 shares has filled 35, a retry for 100 can leave the account with 135 shares. Track cumulative filled quantity and remaining quantity separately. A replacement should target only the outstanding amount, subject to the strategy's current risk limits and any fills received while the replacement request is in flight.
Use an outbox or equivalent durable queue so a process crash between database commit and network submission cannot lose the intent. A worker can pick up pending records, but it must consult the same state machine and reuse the same key.
Add a lease or database lock to limit concurrent submissions for one intent. Leases can expire after a crash; the stable key and reconciliation step still matter.
- Alert on unknown outcomes that remain unresolved beyond the strategy's allowed window.
- Track duplicate-ID responses, ambiguous timeouts, retry counts, late fills and reconciliation mismatches.
- Set account-level limits on open orders, notional exposure and orders per instrument.
- Make a kill switch stop new submissions while leaving status checks and cancellation workflows available.
Paper trading helps exercise process restarts, delayed responses and duplicated messages without risking capital. A controlled agentic paper trading environment such as Agentic Trading Index paper trading can be part of that test set, but it cannot establish how a live broker handles every timeout or deduplication case.
Days 22 to 30: inject failures and release in stages
Test the failure paths deliberately. Drop the response after the broker accepts an order. Delay a response beyond the client timeout.
Return a rate limit, reject an invalid price, restart the worker after saving an intent, and deliver the same queue message twice. Include partial fills followed by a cancel, and a fill that arrives while a replacement is being submitted. For each test, check both the database record and broker-side state.
Use the final days to compare expected and observed outcomes in paper trading, then release in stages. Start with a small instrument set, low notional limits and a human-accessible alert channel. Increase limits only after the system completes a defined number of sessions without unexplained order mismatches. That count is a release criterion you set for your own risk tolerance, not proof that failures are impossible.
Keep operational measures visible: ambiguous outcomes per 1,000 submissions, duplicate orders, time to resolve an unknown state, rejected requests, partial-fill handling errors and maximum exposure during recovery. Review these by broker, strategy and API endpoint. A low average retry count can hide a serious tail case, so inspect individual incidents as well as daily totals.
Stop doing these things:
- Do not retry every timeout as if the order failed.
- Do not generate a fresh client ID for each attempt at the same intent.
- Do not resend the original full quantity after a partial fill.
- Do not clear an unknown order state just because a lookup returned no result once.
Frequently Asked Questions
What does idempotency mean in order placement?
It means repeated submissions for the same trading intent do not create multiple orders. A stable client order ID can support this when the broker enforces deduplication, but the strategy must still track intent and reconcile broker state.
Should an automated trading system retry a timed-out order?
Not until it determines whether the broker accepted the request. Query by client ID and inspect open orders and executions; if the result remains ambiguous and the broker offers no deduplication guarantee, pause and escalate rather than resubmit blindly.
How should retries handle a partially filled order?
Track filled quantity separately from the original order quantity. Any replacement should cover only the remaining quantity, after checking for additional fills and applying current exposure limits.
