Real-Time Alerting for Trading Bots With PagerDuty and Telegram

A bot that silently drifts past its risk limits is more dangerous than one that crashes loudly. When a position breach goes unnoticed for three minutes on a volatile Solana perp, the damage can dwarf the cost of building the alerting pipeline that would have caught it. If you are running automated trading bots in production, a robust alerting layer is not optional — it is part of the core system.

Why Two Channels, Not One

The instinct is to pick Telegram and call it done. Telegram is fast, free, and your phone already has it. The problem is that it has no escalation logic. If you are asleep and a fill anomaly fires at 03:00, a Telegram message sits unread. PagerDuty solves the escalation half: it pages on-call engineers via phone call or SMS, respects quiet hours, rotates between team members, and requires acknowledgment before it stands down. Telegram solves the visibility half: it gives the whole ops team a live stream of bot health that everyone can monitor during market hours.

The right architecture uses both. PagerDuty for anything that requires a human response within minutes, Telegram for ambient observability.

Alert Categories Worth Routing Differently

Not every alert deserves to wake someone up. Classify before you wire:

• P1 — Page immediately: position size exceeds hard limit, account equity drops below margin floor, order fill rate falls to zero for more than 30 seconds, latency spike above 2x rolling baseline sustained for 60 seconds.
• P2 — Telegram only (business hours acknowledgment): slippage above expected model, partial fill rate trending high over a rolling 5-minute window, RPC node latency creeping above 200 ms.
• P3 — Log only: individual order retries, minor fee estimation variance, routine reconnects.

The threshold numbers are specific to your strategy, but the discipline of tiering is universal. Routing everything to PagerDuty trains your team to ignore it.

Wiring PagerDuty: Events API v2

PagerDuty's Events API v2 is the correct integration target for bots. It accepts a JSON payload over HTTPS and handles deduplication via the dedup_key field, which is critical — a bot firing the same alert on every tick will create alert storms that obscure the actual incident.

A minimal Python emit looks like this:

import httpx, time

PD_ROUTING_KEY = "your_integration_key_here"
PD_URL = "https://events.pagerduty.com/v2/enqueue"

def fire_pagerduty(summary: str, severity: str, dedup_key: str, details: dict):
    payload = {
        "routing_key": PD_ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": dedup_key,
        "payload": {
            "summary": summary,
            "severity": severity,          # "critical", "error", "warning", "info"
            "source": "trading-bot",
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "custom_details": details,
        },
    }
    httpx.post(PD_URL, json=payload, timeout=5)

Set dedup_key to something deterministic: f"position_breach_{symbol}_{session_id}". This way, if the bot keeps firing on the same breach, PagerDuty deduplicates and keeps one open incident rather than generating noise. When the condition resolves, send the same dedup_key with "event_action": "resolve". Automatic resolution is worth implementing from day one — stale open incidents erode trust in the system.

Escalation policy configuration: create a service in PagerDuty mapped to one routing key per bot instance. Set a 5-minute acknowledgment timeout before escalating to the secondary on-call. For trading bots running 24/7, the escalation chain matters — if the primary does not acknowledge within 5 minutes, you need a human who will.

Wiring Telegram: Bot API

Telegram alerting is simpler. Create a bot via BotFather, get the token, add the bot to your ops group, and retrieve the chat ID. Then:

import httpx

TG_TOKEN = "your_bot_token"
TG_CHAT_ID = "-100xxxxxxxxxx"   # ops group chat ID

def send_telegram(text: str):
    url = f"https://api.telegram.org/bot{TG_TOKEN}/sendMessage"
    httpx.post(url, json={"chat_id": TG_CHAT_ID, "text": text, "parse_mode": "HTML"}, timeout=5)

Use parse_mode=HTML and wrap key values in <b> tags for scannable messages. A well-formed Telegram alert for a fill anomaly reads: <b>FILL ANOMALY</b> SOL-PERP | Expected 1.2 SOL avg fill, got 0.3 SOL | Slippage 4.1% | 14:32:07 UTC. That gives an engineer everything needed to decide whether to intervene without opening a dashboard.

Keep Telegram messages under 200 characters when possible. Long alerts get truncated on mobile lockscreens, which is exactly when you need to read them.

Shared Alert Queue and Retry Logic

Both channels should read from the same internal queue rather than being called directly from trading logic. This decouples the alert path from the hot path — a PagerDuty timeout should never block an order submission. Use a simple asyncio queue or a lightweight Redis list. Workers consume from the queue and handle retries with exponential backoff capped at 3 attempts. After 3 failures, write to a local dead-letter file and fire a secondary Telegram message from a backup path warning that the primary alert pipeline is degraded.

Never let alerting failure be silent. The meta-alert — "I tried to tell you something and couldn't" — is as important as the original signal.

Testing the Pipeline Before You Need It

Run chaos drills before going live. Inject a synthetic position breach in a staging environment and verify that PagerDuty fires, deduplicates correctly, and auto-resolves. Verify your on-call rotation actually receives the phone call. Send 50 Telegram messages in rapid succession and confirm the bot is not rate-limited by Telegram's 30 messages/second per group ceiling. These are not hypotheticals — you will hit every one of these failure modes eventually, and the correct time to discover them is during a deliberate test, not during a real incident at 02:00.

---

If you want an alerting layer built into your bot from the start rather than bolted on after the first incident, talk to us about what you are running.