5.6 RPCFailoverManager

1. Bot Identity

Operational profile

2. Purpose

Probe RPC providers continuously and fail over before a stale endpoint poisons our chain view.

3. Why This Bot Matters

• Single RPC endpoint goes stale

  All bots reading chain state see an outdated block, causing mispriced or incorrectly-scoped orders.

• No quorum check across providers

  A forked or malicious RPC can poison chain state views used for contract address and balance checks.

• No auto-quarantine of degraded provider

  A slow or error-prone endpoint keeps being polled, adding latency to every on-chain check.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "max_block_lag": 3 }

if (lag >= p.max_block_lag && p.auto_quarantine) quarantine(provider);

{ "min_providers_quorum": 2 }

if (healthyProviders.length < p.min_providers_quorum) return reject('RPC_QUORUM_LOST');

8. Default Configuration

{
  "bot_id": "sec.rpc_failover_manager",
  "version": "0.1.0",
  "mode": "hard_guard",
  "defaults": {
    "max_block_lag": 3,
    "min_providers_quorum": 2,
    "auto_quarantine": true,
    "probe_interval_s": 5
  }
}

9. Implementation Flow

1. On startup: load provider pool from Admin UI config.
2. Background loop every probe_interval_s: call eth_blockNumber on all providers.
3. Compute block height divergence across providers.
4. For providers with lag >= max_block_lag and auto_quarantine=true: mark quarantined.
5. Check healthy provider count; if < min_providers_quorum: REJECT(RPC_QUORUM_LOST) on all pending chain reads.
6. Elect primary provider as the one with highest block height and lowest latency.
7. On order arrival: return current primary provider endpoint for chain reads.
8. Periodically re-probe quarantined providers; restore if lag normalises.

10. Reference Implementation

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. IF/THEN/ELSE = decision. Translate directly to TypeScript, Python, Go, or Rust.

// RPCFailoverManager
STATE = { providers: [], primary: null, quarantined: [] }

// Background probe loop
EVERY params.probe_interval_s:
  heights = []
  FOR provider IN STATE.providers:
    h = FETCH(provider.eth_blockNumber())
    IF h == null: quarantine(provider); CONTINUE
    heights.append({provider, h, latency})
  max_h = MAX(heights.map(x => x.h))
  FOR entry IN heights:
    lag = max_h - entry.h
    IF lag >= params.max_block_lag AND params.auto_quarantine:
      quarantine(entry.provider)
  healthy = heights.filter(x => (max_h - x.h) < params.max_block_lag)
  IF healthy.count < params.min_providers_quorum:
    EMIT RiskVote(DENY, RPC_QUORUM_LOST)
  ELSE:
    STATE.primary = healthy.sort_by(latency).first()
    EMIT RiskVote(APPROVE)

// Per-request provider lookup
FUNCTION getPrimaryProvider():
  IF STATE.primary == null: return DENY(RPC_QUORUM_LOST)
  RETURN STATE.primary

SDK calls used

• provider.eth_blockNumber()
• internal.killswitch.status()

Complexity: O(p) per probe where p = provider pool size (small constant)

11. Wire Examples

Input — what arrives on the wire

Probe result from provider pool — onchain

{
  "provider": "alchemy-polygon-1",
  "block_number": 58420100,
  "latency_ms": 45,
  "timestamp_ms": 1746768672000
}

Output — what the bot emits

RiskVote — APPROVE with primary

{
  "vote_id": "sec.rpc_failover_manager.20260509T170000Z",
  "decision": "APPROVE",
  "reason_code": null,
  "evidence": {
    "primary_provider": "alchemy-polygon-1",
    "healthy_count": 3,
    "max_lag_blocks": 1
  },
  "checked_at": "2026-05-09T17:00:00Z"
}

12. Decision Logic

13. Standard Decision Output

This bot returns a RiskVote object. See RiskVote schema.

{
  "vote_id": "sec.rpc_failover_manager.20260509T170000Z",
  "decision": "APPROVE",
  "reason_code": null,
  "evidence": {
    "primary_provider": "alchemy-polygon-1",
    "healthy_count": 3,
    "quarantined_count": 0,
    "max_lag_blocks": 1
  },
  "checked_at": "2026-05-09T17:00:00Z"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

16. Developer Reporting

{
  "bot_id": "sec.rpc_failover_manager",
  "decision": "APPROVE",
  "inputs_used": [
    "onchain.eth_blockNumber",
    "config.provider_pool"
  ],
  "checked_at": "2026-05-09T17:00:00Z"
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

Re-probe all providers on restart; no persistent state required.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

Sibling bots (same OrderIntent)

Used by (auto-aggregated)

5.2 5.7

External services

23. Security Surfaces

Abuse vectors considered

• Compromised RPC provider returning fraudulent block heights to pass quorum
• BGP hijack routing traffic to malicious RPC node

Mitigations

• Quorum of min_providers_quorum providers required; single provider cannot pass unilaterally
• Provider pool configured in Admin UI with TLS-pinned endpoints

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

RPCQuorumLost is a P0 event — all chain-dependent checks are blocked. Restore provider connectivity immediately.

On-call actions

Manual overrides

• —

Healthcheck

GET /internal/health/rpcfailovermanager → green if At least min_providers_quorum healthy providers; primary elected within last probe_interval_s.; red if healthy_count < min_providers_quorum or no probe completed in last 30s.

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%