Monitoring a Solana Trading Bot with Prometheus and Grafana

Running a Solana trading bot in production without metrics is flying blind. You will not notice that your RPC node is 40 slots behind until your fills stop matching or your arb leg executes at a stale price. This guide covers exactly how we instrument bots at TierZero — what to measure, how to expose it, and how to build dashboards that surface real problems before they become expensive ones.

What to Measure and Why

Four metric families cover the majority of failure modes in a Solana bot:

• Slot lag — the difference between the slot your RPC reports as latest and the actual cluster head. Anything above 5 slots on mainnet-beta is a yellow flag; above 15 is a red one. You are trading on stale state.
• RPC error rate — 429 Too Many Rate Limited, 503 Service Unavailable, and JSON-RPC ServerError -32005 all indicate your connection is degraded. Even brief spikes cause missed fills.
• Fill latency — time from order submission (sendTransaction) to confirmed slot. On a healthy validator with a well-constructed transaction, this should land between 400 ms and 800 ms. Consistent outliers above 1.5 s point to transaction propagation issues or priority fee miscalculation.
• Realized PnL per cycle — not for vanity; for detecting strategy drift. If PnL per trade drops 30% over two hours without a market explanation, something is wrong in your signal or execution path.

Exposing Metrics from TypeScript

The prom-client library handles Prometheus exposition in Node.js with minimal boilerplate. Register your metrics once at module load, then update them inside your trading loop.

import { Gauge, Counter, Histogram, Registry } from 'prom-client';

const registry = new Registry();

const slotLag = new Gauge({
  name: 'bot_slot_lag',
  help: 'Slots behind cluster head',
  registers: [registry],
});

const rpcErrors = new Counter({
  name: 'bot_rpc_errors_total',
  help: 'RPC errors by type',
  labelNames: ['error_code'],
  registers: [registry],
});

const fillLatency = new Histogram({
  name: 'bot_fill_latency_seconds',
  help: 'sendTransaction to confirmed slot',
  buckets: [0.2, 0.5, 1.0, 1.5, 3.0, 10.0],
  registers: [registry],
});

const pnlGauge = new Gauge({
  name: 'bot_realized_pnl_usd',
  help: 'Cumulative realized PnL in USD',
  registers: [registry],
});

Expose the /metrics endpoint on a dedicated port (we use 9090 internally, but keep it firewalled — Prometheus scrapes it from within the same private network):

import http from 'http';

http.createServer(async (req, res) => {
  if (req.url === '/metrics') {
    res.setHeader('Content-Type', registry.contentType);
    res.end(await registry.metrics());
  }
}).listen(9090);

Inside your main loop, record the values after every RPC call and after every fill confirmation. Track slot lag by comparing getSlot() responses to a reference node you trust — ideally a dedicated validator or a second RPC provider.

Exposing Metrics from Rust

In Rust, metrics + metrics-exporter-prometheus give you a similar pattern without the overhead of a full async runtime just for telemetry:

use metrics::{counter, gauge, histogram};
use metrics_exporter_prometheus::PrometheusBuilder;

PrometheusBuilder::new()
    .with_http_listener(([127, 0, 0, 1], 9090))
    .install()
    .expect("failed to install Prometheus exporter");

// Inside your trading loop:
gauge!("bot_slot_lag", lag as f64);
counter!("bot_rpc_errors_total", 1, "error_code" => code.to_string());
histogram!("bot_fill_latency_seconds", latency_secs);

The Rust exporter handles the HTTP server in a background thread. Keep cardinality low on label dimensions — do not use transaction signatures as label values or you will OOM Prometheus within hours.

Prometheus Scrape Config and Storage

A minimal prometheus.yml scrape block:

scrape_configs:
  - job_name: 'trading_bot'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Five-second scrape intervals are fine for trading bots. One-second resolution exists if you need it, but at that cadence you are better off looking at your logging pipeline rather than metrics. For retention, 15 days at 5 s resolution for a single bot costs roughly 500 MB on disk — negligible.

Grafana Dashboard Layout

Build your dashboard in three rows:

Row 1 — RPC Health. bot_slot_lag as a time-series panel with a red threshold line at 15. rate(bot_rpc_errors_total[1m]) as a stacked bar by error_code. If either panel lights up, the on-call engineer knows where to look in under ten seconds.

Row 2 — Execution Quality. histogram_quantile(0.95, rate(bot_fill_latency_seconds_bucket[5m])) for p95 fill latency. Target: below 800 ms. If the p95 drifts above 1.5 s, your priority fees are too low or your TPU routing is degraded.

Row 3 — Strategy Performance. bot_realized_pnl_usd as a cumulative line. Add a calculated field showing PnL delta over the last 30 minutes. A flat or declining line during active market hours is an immediate investigation trigger.

Alert Rules That Actually Fire on the Right Things

Alertmanager routing is where most teams get lazy. These three rules cover the highest-value failure modes:

groups:
  - name: bot_alerts
    rules:
      - alert: HighSlotLag
        expr: bot_slot_lag > 15
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "RPC node is {{ $value }} slots behind cluster head"

      - alert: ElevatedRpcErrors
        expr: rate(bot_rpc_errors_total[2m]) > 0.5
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "RPC error rate {{ $value | humanize }}/s"

      - alert: FillLatencyDegraded
        expr: histogram_quantile(0.95, rate(bot_fill_latency_seconds_bucket[5m])) > 1.5
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "p95 fill latency {{ $value | humanizeDuration }}"

The for duration on HighSlotLag is deliberately short — 30 seconds of high lag on a latency-sensitive strategy is already meaningful loss. For the error rate alert, a two-minute burn window filters out transient blips without letting real degradation hide.

One pattern worth the extra work: push bot_realized_pnl_usd deltas to a dead-man's-switch alert. If the metric has not changed in 10 minutes during expected market hours, the bot has stopped trading — whether from an exception, a stale connection, or a logic bug. That alert has caught more silent failures for us than any other single rule.

---

If you want a bot that ships with production-grade observability out of the box, talk to us — instrumentation, dashboards, and alert routing are part of every engagement we run.