6.2 Health & Heartbeat

1. Bot Identity

2. Purpose

HealthHeartbeat monitors the liveness of all 97 production bots by polling each bot's internal health endpoint at a configurable interval. If a bot misses missed_heartbeats_to_alert consecutive polls, HealthHeartbeat emits a page-severity alert and optionally triggers an auto-restart. It emits an OperationsReport after every sweep cycle summarising bot health across all layers. Internal-only — no external API surface.

3. Why This Bot Matters

• A bot crashes silently without HealthHeartbeat running

  The dead bot's layer is unguarded. Risk votes, kill-switch checks, or execution guards may stop firing, allowing uncontrolled order flow.

• Auto-restart fires for a bot in a crash-loop

  Repeated restarts mask a systemic failure and exhaust restart budgets. Without a circuit breaker, the governance layer itself degrades.

• Alert not fired on missed heartbeats

  On-call is not paged. The dead bot may go unnoticed for hours, accumulating unmonitored risk exposure.

• HealthHeartbeat itself is not monitored

  The watchdog is unwatched. A dead HealthHeartbeat means all 97 bots run without liveness supervision.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "heartbeat_interval_s": 30 }

if (p.heartbeat_interval_s > p.hard) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL')

{ "missed_heartbeats_to_alert": 3 }

if (p.missed_heartbeats_to_alert > p.hard) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL')

{ "auto_restart": true }

if (p.auto_restart && misses >= p.missed_heartbeats_to_alert) triggerRestart(bot_slug)

{ "page_on_failure": true }

if (!p.page_on_failure) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL')

8. Default Configuration

{
  "bot_id": "gov.health_heartbeat",
  "version": "2.0.0",
  "mode": "general_live",
  "defaults": {
    "heartbeat_interval_s": 30,
    "missed_heartbeats_to_alert": 3,
    "auto_restart": true,
    "page_on_failure": true
  },
  "locked": {
    "page_on_failure": {
      "immutable": true
    },
    "heartbeat_interval_s": {
      "max": 300
    },
    "missed_heartbeats_to_alert": {
      "max": 10
    }
  }
}

9. Implementation Flow

1. On startup, load the bot registry from the config store; build a polling table keyed by bot_slug with miss_count=0.
2. Every heartbeat_interval_s, iterate over all registered bots and issue GET /internal/health/<slug> with a timeout of heartbeat_interval_s/3.
3. For each bot: if response is 200 within timeout, reset miss_count to 0 and emit INFO heartbeat.
4. If response is non-200 or times out, increment miss_count.
5. When miss_count >= missed_heartbeats_to_alert: emit page alert (HEALTH_HEARTBEAT_BOT_DOWN) and, if auto_restart=true, publish restart command to the process manager.
6. Track restart budget per bot (default 3 restarts per 10 minutes). If budget is exhausted, emit HEALTH_HEARTBEAT_RESTART_BUDGET_EXHAUSTED and stop auto-restarting.
7. After each full sweep, emit an OperationsReport summarising: total_bots, healthy_count, unhealthy_count, restarted_count, sweep_duration_ms.
8. HealthHeartbeat itself is monitored by a watchdog process (deadman timer) that pages if no OperationsReport is emitted within 2x heartbeat_interval_s.

10. Reference Implementation

Polls all 97 registered bots' health endpoints every heartbeat_interval_s, tracks consecutive misses, fires alerts and auto-restarts at threshold, emits a sweep OperationsReport after each cycle.

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. Translate to TS/Python/Go/Rust.

// ---- STARTUP ----
FUNCTION init():
  registry = FETCH config_store.GET('/bot-registry')
  miss_counts = { slug: 0 FOR slug IN registry }
  restart_budgets = { slug: { count: 0, window_start: now() } FOR slug IN registry }
  setInterval(runSweep, config.heartbeat_interval_s * 1000)

// ---- SWEEP ----
FUNCTION runSweep():
  sweep_start = now()
  healthy = 0; unhealthy = 0; restarted = 0
  unhealthy_bots = []

  FOR bot IN registry:
    response = FETCH GET '/internal/health/' + bot.slug
      TIMEOUT config.heartbeat_interval_s / 3 * 1000

    IF response.status == 200:
      IF miss_counts[bot.slug] >= config.missed_heartbeats_to_alert:
        EMIT alert(HEALTH_HEARTBEAT_BOT_RECOVERED, bot.slug)
      miss_counts[bot.slug] = 0
      healthy += 1
    ELSE:
      miss_counts[bot.slug] += 1
      unhealthy += 1

      IF miss_counts[bot.slug] >= config.missed_heartbeats_to_alert:
        alerting.emit('HEALTH_HEARTBEAT_BOT_DOWN', {
          slug: bot.slug, miss_count: miss_counts[bot.slug] })

        IF config.auto_restart:
          budget = restart_budgets[bot.slug]
          IF (now() - budget.window_start) > 600_000:  // 10-min window
            budget.count = 0; budget.window_start = now()
          IF budget.count < 3:
            internal_bus.publish('process.restart', { slug: bot.slug })
            budget.count += 1; restarted += 1
            unhealthy_bots.append({ slug: bot.slug, miss_count: miss_counts[bot.slug], action: 'restarted' })
          ELSE:
            alerting.emit('HEALTH_HEARTBEAT_RESTART_BUDGET_EXHAUSTED', { slug: bot.slug })
            unhealthy_bots.append({ slug: bot.slug, miss_count: miss_counts[bot.slug], action: 'budget_exhausted' })

  EMIT OperationsReport({
    report_id:         'ops_health_' + sweep_start,
    event_type:        'HEALTH_SWEEP_COMPLETE',
    total_bots:        len(registry),
    healthy_count:     healthy,
    unhealthy_count:   unhealthy,
    restarted_count:   restarted,
    sweep_duration_ms: now() - sweep_start,
    unhealthy_bots:    unhealthy_bots,
    fired_at_ms:       sweep_start
  })

SDK calls used

• config_store.GET('/bot-registry')
• FETCH GET '/internal/health/<slug>' TIMEOUT <ms>
• internal_bus.publish('process.restart', { slug })
• alerting.emit('HEALTH_HEARTBEAT_BOT_DOWN', metadata)
• alerting.emit('HEALTH_HEARTBEAT_RESTART_BUDGET_EXHAUSTED', metadata)

Complexity: O(N) per sweep where N = 97 registered bots

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Health endpoint poll response",
  "source": "internal /internal/health/<slug>",
  "payload": {
    "slug": "strat.some_strategy",
    "status": "ok",
    "last_decision_ms": 1746791970000,
    "uptime_s": 86400
  }
}

Output — what the bot emits

{
  "label": "OperationsReport — HEALTH_SWEEP_COMPLETE",
  "payload": {
    "report_id": "ops_health_1746792000000",
    "bot_id": "gov.health_heartbeat",
    "event_type": "HEALTH_SWEEP_COMPLETE",
    "total_bots": 97,
    "healthy_count": 96,
    "unhealthy_count": 1,
    "restarted_count": 1,
    "sweep_duration_ms": 840,
    "unhealthy_bots": [
      {
        "slug": "strat.some_strategy",
        "miss_count": 3,
        "action": "restarted"
      }
    ],
    "fired_at_ms": 1746792000000,
    "report_kind": "OperationsReport"
  }
}

12. Decision Logic

13. Standard Decision Output

This bot returns a OperationsReport object. See OperationsReport schema.

{
  "report_id": "ops_health_20260509T120000Z",
  "bot_id": "gov.health_heartbeat",
  "event_type": "HEALTH_SWEEP_COMPLETE",
  "total_bots": 97,
  "healthy_count": 96,
  "unhealthy_count": 1,
  "restarted_count": 1,
  "sweep_duration_ms": 840,
  "unhealthy_bots": [
    {
      "slug": "strat.some_strategy",
      "miss_count": 3,
      "action": "restarted"
    }
  ],
  "fired_at_ms": 1746792000000,
  "report_kind": "OperationsReport"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

Dashboards

• Grafana — Governance / HealthHeartbeat liveness overview (all 97 bots)
• Grafana — Governance / Auto-restart rate and budget consumption

16. Developer Reporting

{
  "bot_id": "gov.health_heartbeat",
  "event_type": "HEALTH_BOT_MISS",
  "slug": "strat.some_strategy",
  "miss_count": 2,
  "threshold": 3,
  "last_seen_ms": 1746791940000,
  "fired_at_ms": 1746791970000
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

On restart, all miss_counts reset to 0. The first sweep re-establishes the health baseline.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

Sibling bots (same OrderIntent)

External services

23. Security Surfaces

Abuse vectors considered

• A bot returns a fake 200 response from a zombie process to avoid restart
• Raising missed_heartbeats_to_alert to a very high value to prevent alerts from firing
• Disabling page_on_failure to suppress alerting

Mitigations

• page_on_failure is locked immutable; cannot be disabled
• heartbeat_interval_s and missed_heartbeats_to_alert have hard maximums enforced at config load
• Health endpoint responses are checked for a valid JSON body, not just HTTP status
• HealthHeartbeat itself is monitored by an external deadman watchdog

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

HealthHeartbeat incidents are either a bot going down (most common), the restart budget exhausting on a crash-looping bot, or HealthHeartbeat itself failing. All three require immediate response.

On-call actions

Manual overrides

• polytraders gov health pause-restart --slug <slug> — Stop auto-restart for a specific bot while investigating a crash-loop.

Healthcheck

Endpoint: /internal/health/health-heartbeat | Green: Last sweep completed within 2x heartbeat_interval_s; all bots polled; OperationsReport emitted. | Red: No sweep in 2x heartbeat_interval_s; registry load failed; process unresponsive.

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%