Trading Bot Order State Machines: Idempotent Fills & Crash Recovery

A bot that placed a Hyperliquid order, got a socket timeout, restarted, and placed the same order again just cost its owner a double position. That failure mode has almost nothing to do with your strategy and everything to do with the fact that "send an order" is not one operation — it's a lifecycle with at least six distinct states, any of which can be interrupted by a crash, an RPC drop, or an OOM kill. If you don't model that lifecycle explicitly and persist it, your bot's correctness is a coin flip every time the process dies mid-flight.

Most tutorials stop at transaction-level retries: resubmit the Solana tx, bump the priority fee, done. That solves confirmation, not intent. The harder problem is making order placement itself exactly-once across process restarts, and that requires a persisted state machine sitting one layer above the RPC client.

Why tx-level retry isn't enough

Retrying a Solana transaction is safe because the blockchain deduplicates by signature — the same signed tx can hit the network 20 times and land once. But that guarantee only holds if you reuse the same signed transaction with the same blockhash. The moment your bot restarts and rebuilds the order from strategy state, you've minted a new signature, and the network will happily land both.

Centralized venues are worse. Hyperliquid's exchange endpoint has no native idempotency key on the info side, so a POST that times out on the client leaves you genuinely unsure whether the fill happened. Polymarket's CLOB returns an order ID only on success; a dropped response means you have an order living on the book that your local state has never heard of. In every case the fix is the same: assign a client-side identity to the order intent before you ever touch the wire, and make the whole path idempotent on that identity.

The lifecycle as an explicit state machine

Model an order as a row that moves through these states, and never skips:

• PENDING — intent created, nothing sent
• SUBMITTED — request left the process, outcome unknown
• ACKED — venue returned an order ID / signature
• PARTIAL — some quantity filled
• FILLED / CANCELED / REJECTED — terminal

The critical states are SUBMITTED and ACKED. SUBMITTED is the danger zone — you've committed a side effect but don't know its result. Recovery logic lives entirely in reconciling rows stuck in SUBMITTED.

Two rules make the machine safe:

1. Persist before the side effect. Write SUBMITTED to durable storage and fsync (or let your DB's WAL do it) before the network call, not after. If you crash after writing but before sending, recovery re-sends against a stable client ID and the venue dedupes. If you crash after sending but before writing, you've broken exactly-once — so the write must come first.
2. Every transition is a compare-and-set. UPDATE orders SET state='ACKED' WHERE id=? AND state='SUBMITTED'. If the affected row count is zero, another path already advanced it. This is what stops two coroutines — the submit path and the reconciler — from both acting on the same order.

Getting these two rules right is the bulk of the real work, and it's the part that's cheapest to get wrong on your own; a short strategy consultation before you commit to a persistence layer usually pays for itself.

The client order ID is your idempotency key

Generate a deterministic ID at intent time and thread it through every venue that will accept one.

import hashlib, time

def client_order_id(strategy: str, symbol: str, epoch_ms: int, nonce: int) -> str:
    # deterministic: same inputs on replay -> same id
    raw = f"{strategy}:{symbol}:{epoch_ms}:{nonce}".encode()
    return "cl_" + hashlib.blake2s(raw, digest_size=12).hexdigest()

async def place(order):
    # 1. persist intent FIRST
    await db.upsert_order(order.id, state="SUBMITTED", cloid=order.cloid)
    try:
        resp = await venue.submit(order, cloid=order.cloid)  # idempotent on cloid
        await db.cas(order.id, "SUBMITTED", "ACKED", venue_id=resp.oid)
    except (Timeout, ConnectionError):
        # leave it SUBMITTED; reconciler will resolve it
        pass

Venue specifics matter:

• Hyperliquid accepts a cloid (128-bit hex) on orders. On restart you query open orders and recent fills filtered by cloid; if it's there, you skip the resubmit. This turns "did my order land?" into a lookup instead of a guess.
• Jito bundles dedupe on bundle content and expose a bundle UUID, but a landed bundle and a dropped getBundleStatuses response look identical to a naive client. Persist the bundle ID at SUBMITTED and poll status on recovery before rebuilding. If you're sniping, this reconciliation gap is exactly where duplicate buys come from — worth understanding deeply before you ship a Solana sniper bot.
• Polymarket CLOB signs each order with a salt; reuse the same salt on replay and the same signed order produces the same hash, so a resubmit is a no-op if the first one landed. The neg-risk mechanics get subtle fast, and we walked through the adjacent split/merge edge cases in the neg-risk adapter arbitrage piece.

Recovery: the reconciliation loop

On boot, before the strategy runs a single tick, load every non-terminal order and reconcile:

for order in db.non_terminal_orders():
    truth = venue.lookup(cloid=order.cloid)   # source of truth = venue
    if truth.exists:
        db.cas(order.id, order.state, truth.state, venue_id=truth.oid)
    elif order.state == "SUBMITTED" and order.age < resubmit_ttl:
        resubmit(order)          # same cloid, venue dedupes
    else:
        db.cas(order.id, order.state, "REJECTED")  # give up, alert

The venue is always the source of truth; your local row is a cache that can lag. The resubmit_ttl guards against re-sending a stale intent whose price is now wrong — a 5-second-old sniper order is worthless, so you reject rather than resubmit. Market makers invert this: a resting quote that dropped off the book should be re-placed, which is why the Hyperliquid market maker reconciliation TTL is measured in minutes, not seconds.

Gotchas that bite in production

• Partial fills during the gap. If you crash while PARTIAL, recovery must read filled quantity from the venue, not assume zero. Track filled_qty as a column and reconcile the remainder.
• Clock skew on deterministic IDs. If epoch_ms comes from wall-clock and NTP jumps on restart, your "deterministic" replay generates a new cloid and defeats dedup. Snapshot the intent's timestamp into the persisted row and replay from that, never from now().
• Terminal-state finality. Once a row is FILLED, no path may reopen it. Enforce it in the CAS: transitions out of terminal states should affect zero rows.

The same discipline scales down to a single Polymarket spread bot and up to a multi-venue book, which is why we bake the state machine into every Polymarket spread bot build rather than treating recovery as an afterthought. If you're scoping this out, the section on failure modes in our guide to writing a trading bot spec is where the crash-recovery requirements should live from day one.

Get the persistence-before-side-effect ordering right and the rest is mechanical. If you'd rather have that machine designed correctly the first time instead of debugging double-fills at 3am, that's exactly what our strategy consultation is for.