Incident Response Playbook for Crypto Trading Bot Failures

A crypto trading bot incident response playbook is not something most shops write until they need one. That is too late. When a bot starts legging into positions it was never supposed to take — or mis-sizes an order by 100x because a WebSocket reconnection delivered a stale fill — you have seconds to decide who acts, in what order, and what to preserve before evidence disappears into log rotation. This runbook covers the first 15 minutes.

Immediate Kill Chain (0–2 Minutes)

The person who first notices something is wrong has exactly one job: stop the bleeding, not diagnose it. Diagnosis comes later.

Step 1 — Kill the bot process first. Send SIGTERM so the process has a chance to cancel open orders via the exchange API. Wait 5 seconds. If it has not exited, SIGKILL. Do not try to "gently shut down" a misbehaving bot — the risk of it placing more orders while you're reading logs is not worth it.

Step 2 — Cancel all open orders manually. On Hyperliquid, use the /cancelAll endpoint or the web UI; on Solana DEXes with on-chain order books (e.g. OpenBook), submit a cancel transaction directly from your wallet, not via the bot. Assume the bot's own cancel logic is compromised.

Step 3 — Pause or hedge, don't immediately flatten. If the bot is deeply into a position and the market is moving against you, a panicked market-order flatten can be far more expensive than a limit-order exit or a quick hedge. Decide in the first 60 seconds: can you delta-hedge the risk to buy time? A flat hedge via a correlated perp buys you 10 minutes to make a clean decision.

Assign these three steps to named roles before any incident occurs. Operator (process kill), Trader (order cancel + hedge decision), Observer (start recording everything). If you are a one-person shop, do them strictly in sequence.

Evidence Preservation (2–5 Minutes)

You will want to do a post-mortem. Do not let logs rotate or in-memory state get cleared before you capture it.

• Copy bot logs to a timestamped file immediately. Most production bots run under systemd or Docker. Run docker logs --since 1h <container> > incident-$(date +%s).log before you restart anything.
• Snapshot the process's open file descriptors and memory-mapped files if you suspect a state-corruption bug. On Linux: ls -la /proc/<pid>/fd and consider a core dump before kill.
• Export exchange state. Pull your open positions, fills and order history from the exchange API right now, before you start cancelling things and losing the before/after diff. On Hyperliquid: GET /info with type: clearinghouseState and type: userFills for the last 24 hours. On Solana: grab the relevant transaction signatures from your wallet address via the RPC method getSignaturesForAddress — you want the last 200 signatures with commitment: confirmed.
• Screenshot the UI if you use one. Block explorers sometimes lag; your dashboard's current state is faster to read in a post-mortem than reconstructing it from logs.

The goal here is a snapshot of the world at time-of-incident, not a cleaned-up narrative.

Auditing the Trade History on Solana and Hyperliquid

Once the bot is stopped and positions are hedged, reconstruct exactly what happened.

Solana. Use getSignaturesForAddress with your bot's fee-payer public key, then fetch each transaction with getTransaction using maxSupportedTransactionVersion: 0. Parse the inner instructions — Jupiter and Raydium wrap swap logic several layers deep and the top-level instruction will not tell you the actual fill price. The preBalances and postBalances (or preTokenBalances / postTokenBalances) fields give you the exact amounts in and out. Tools like Solana FM and Step Finance can reconstruct a human-readable trade history from a wallet address in under a minute.

Hyperliquid. Pull userFills from the info endpoint with a broad time window. Each fill record includes the oid (order ID), side, sz, px, fee, and a crossed boolean that tells you whether you were the taker. Cross-reference these against your bot's internal order log — a mismatch between what the bot thought it submitted and what actually filled is usually the root cause you are looking for. Hyperliquid also exposes userFunding which matters if your incident involved a perp position that ran for longer than one funding interval.

Reproduce the event timeline in a spreadsheet: timestamp, order submitted, fill received, position delta, cumulative exposure. This is the artifact your post-mortem runs off.

Incident Response Playbook: The 15-Minute Checklist

Keep this as a pinned message in your ops Telegram channel or a Notion page your whole team can reach without logging into any system.

Minutes 0–2

• Bot process killed (SIGTERM → SIGKILL)
• Open orders cancelled via exchange UI/API directly
• Hedge placed or flatten decision made

Minutes 2–5

• Logs exported to timestamped file
• Exchange fill history pulled via API and saved
• Solana transaction signatures captured (if applicable)
• Dashboard/UI screenshot taken

Minutes 5–10

• Current P&L vs expected P&L calculated — quantify the damage
• Root cause hypothesis formed (logic bug, stale data, reconnect issue, config error)
• Restart decision made: is the bug understood well enough to restart safely?

Minutes 10–15

• LP or counterparty communication sent if necessary (see below)
• Post-mortem ticket created with timeline, evidence links and hypothesis
• Monitoring alerts reviewed — did anything fire before the human noticed?

Communicating with Counterparties and LPs

If you run capital on behalf of LPs, or if your bot operates inside a liquidity partnership (e.g. a token team's MM arrangement), you have an obligation to communicate quickly and factually.

What to send within 15 minutes: "We experienced an automated trading incident at [time UTC]. The bot has been stopped. Positions are [closed / hedged]. We are investigating. Estimated loss is approximately $X. We will send a full post-mortem within 24 hours."

What not to send: speculation about cause, blame, or reassurances that "it won't happen again" before you know why it happened.

If the incident involved a Polymarket position hitting resolution incorrectly, or a market-making arrangement where your sudden absence widened spreads abnormally, note that explicitly — it affects the counterparty's own records and they will appreciate the heads-up.

Keep the initial message short. LPs do not want a technical essay; they want to know you are in control and that you will give them the full picture later.

Designing Bots That Fail Gracefully

The best incident response is the one you never have to run because the bot protected itself. Every trading bot we build at TierZero ships with at minimum: a kill-switch that can be triggered via Telegram command, hard per-order and per-session exposure caps enforced in code (not just config), and structured logging that writes order state to disk before submitting to the exchange. See our trading dashboard and control panel service for how we build the operational layer — kill-switches, live P&L, and alerting — as a first-class deliverable rather than an afterthought.

When you design a bot, design its failure modes first. What does it do when the WebSocket drops mid-fill? What does it do when the RPC returns a stale slot? What does it do when it receives two fills for the same order ID? Every one of these scenarios has a safe answer that costs almost nothing to implement upfront and an expensive answer that costs a lot to clean up in production.

The kill-switch should always be the simplest, most reliable path in the system — not the most automated one.

---

If you are evaluating bot infrastructure or need a review of your existing system's failure modes, get in touch. We have shipped production bots on Solana, Hyperliquid and Polymarket and have run through enough real incidents to know where the bodies are buried.