6.11 IncidentCommander

1. Bot Identity

2. Purpose

IncidentCommander coordinates halts, flattens, and post-mortems when a guard, monitor, or operator declares an incident. It records the incident timeline, dispatches auto-actions by severity, pages on-call, and tracks RCA completion.

3. Why This Bot Matters

• No centralised incident coordinator

  Multiple bots may take conflicting halt/flatten actions; incident timeline is incoherent.

• RCA not completed within SLA

  Repeat incidents occur because root cause is never addressed; compliance audit finds gaps.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "auto_actions_by_severity": {"P0": ["halt_all", "page_oncall"], "P1": ["page_oncall"], "P2": ["notify_slack"]} }

actions = p.auto_actions_by_severity.get(incident.severity, [])

{ "require_rca_within_h": 24 }

if now() - incident.resolved_at > hours(p.require_rca_within_h): emit('RCA_OVERDUE')

8. Default Configuration

{
  "bot_id": "gov.incidentcommander",
  "version": "0.1.0",
  "mode": "shadow_only",
  "defaults": {
    "auto_actions_by_severity": {
      "P0": [
        "halt_all",
        "page_oncall"
      ],
      "P1": [
        "page_oncall"
      ],
      "P2": [
        "notify_slack"
      ]
    },
    "require_rca_within_h": 24,
    "page_on_severity": "P1",
    "publish_status_externally": false
  }
}

9. Implementation Flow

1. On incident declaration, assign incident_id (ULID) and record severity, scope, and declaring bot.
2. Dispatch auto-actions from auto_actions_by_severity map for the declared severity.
3. Page on-call if incident severity >= page_on_severity threshold.
4. Record incident timeline events: declaration, auto-actions taken, acknowledgement, resolution.
5. After resolution, start require_rca_within_h countdown; emit RCA_OVERDUE if RCA is not filed in time.
6. Emit OperationsReport(event_type=INCIDENT_DECLARED/RESOLVED/RCA_FILED) on each lifecycle event.

10. Reference Implementation

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. IF/THEN/ELSE = decision. Translate directly to TypeScript, Python, Go, or Rust.

// ---- INCIDENT DECLARATION ----
FUNCTION declareIncident(declaration):
  incident = {
    id: generateULID(), severity: declaration.severity,
    scope: declaration.scope, declared_by: declaration.bot_id,
    declared_at: now(), status: 'active', timeline: []
  }
  postgres.insert('incidents', incident)
  actions = config.auto_actions_by_severity.get(incident.severity, [])
  FOR action IN actions:
    dispatch(action, incident)
    incident.timeline.append({action: action, at: now()})
  EMIT OperationsReport(event_type='INCIDENT_DECLARED', incident_id=incident.id,
    severity=incident.severity, auto_actions_dispatched=actions)

// ---- RESOLUTION ----
FUNCTION resolveIncident(incidentId, resolvedBy):
  incident = postgres.get('incidents', incidentId)
  incident.status = 'resolved'
  incident.resolved_at = now()
  incident.resolved_by = resolvedBy
  postgres.upsert('incidents', incident)
  scheduleRcaDeadline(incidentId, config.require_rca_within_h)
  EMIT OperationsReport(event_type='INCIDENT_RESOLVED', incident_id=incidentId)

// ---- RCA CHECK ----
FUNCTION checkRcaDeadline(incidentId):
  incident = postgres.get('incidents', incidentId)
  IF incident.rca_filed IS NULL:
    EMIT OperationsReport(event_type='RCA_OVERDUE', incident_id=incidentId)
    alerting.emit('RCA_OVERDUE', {incident_id: incidentId})

SDK calls used

• postgres.insert('incidents', incident)
• postgres.upsert('incidents', incident)
• alerting.emit('RCA_OVERDUE', metadata)

Complexity: O(1) per incident event; O(A) per declaration where A = auto-action count

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Incident declaration",
  "source": "risk.liquidityguard",
  "payload": {
    "declaring_bot": "risk.liquidityguard",
    "severity": "P1",
    "scope": [
      "exec.smartrouter"
    ],
    "declared_at": "2026-05-09T10:00:00Z"
  }
}

Output — what the bot emits

{
  "label": "OperationsReport — INCIDENT_DECLARED",
  "payload": {
    "report_id": "ops_incident_01HX9Z",
    "event_type": "INCIDENT_DECLARED",
    "incident_id": "inc_01HX9Z",
    "severity": "P1",
    "report_kind": "OperationsReport",
    "topic": "polytraders.reports.operations"
  }
}

12. Decision Logic

13. Standard Decision Output

This bot returns a OperationsReport object. See OperationsReport schema.

{
  "report_id": "ops_incidentcommander_01HX9Z",
  "bot_id": "gov.incidentcommander",
  "event_type": "INCIDENT_DECLARED",
  "incident_id": "inc_01HX9Z",
  "severity": "P1",
  "scope": [
    "risk.liquidityguard",
    "exec.smartrouter"
  ],
  "auto_actions_dispatched": [
    "page_oncall"
  ],
  "declared_at": "2026-05-09T10:00:00Z",
  "report_kind": "OperationsReport",
  "topic": "polytraders.reports.operations"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

16. Developer Reporting

{
  "bot_id": "gov.incidentcommander",
  "event_type": "AUTO_ACTION_DISPATCHED",
  "incident_id": "inc_01HX9Z",
  "action": "page_oncall",
  "dispatched_at_ms": 1746792060000
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

On restart, reload active incidents from Postgres; re-schedule RCA deadlines.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

Sibling bots (same OrderIntent)

External services

23. Security Surfaces

Abuse vectors considered

• Declaring a spurious P0 incident to trigger halt_all and disrupt trading

Mitigations

• Incident declarations require an authenticated internal bot or operator identity
• Incident timeline is immutably logged; false declarations are auditable

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

IncidentCommander incidents require immediate triage. P0 auto-actions halt trading; confirm the incident is genuine before clearing.

On-call actions

Manual overrides

• —

Healthcheck

/internal/health/incidentcommander → green if No active P0 incidents; paging system reachable; no RCA overdue; red if Active P0 incident or paging system unreachable

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%