6.4 Backtester

1. Bot Identity

2. Purpose

Backtester replays historical CLOB snapshots and the full report archive through the live execution path at tick resolution. It runs in replay mode (mode=replay), consuming archived ObservationReport, DecisionReport, RiskVote, ExecutionReport, and SettlementReport envelopes and re-executing the strategy under test against them. It emits replay-tagged OperationsReport records (and replay-tagged copies of every simulated report kind) to polytraders.reports.operations, partitioned by bot_slug+epoch, retained for 1 year. Backtester is used by the governance team to validate parameter changes, A/B test strategies, and produce audit-quality evidence before any strategy is promoted to live. Backtester never signs orders or touches the live CLOB.

3. Why This Bot Matters

• Strategy promoted to live without backtesting

  Untested parameter changes or new strategy logic may produce runaway losses or adverse fills on live capital. Backtesting is a mandatory gate before promotion.

• Backtester uses different execution path than live

  Results are not comparable to live performance. Governance audit evidence is invalid and cannot be used to justify promotion decisions.

• Replay-tagged reports not emitted

  Backtesting runs are not auditable. Governance cannot produce the required evidence trail for strategy promotion gates.

• Parameter sweep runs with non-deterministic inputs

  Results are not reproducible. Successive backtests on the same window may produce different outputs, making comparison and audit impossible.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "start_ts": "now()-7d" }

if (p.start_ts >= p.end_ts) throw ConfigError('BACKTESTER_INVALID_WINDOW')

{ "end_ts": "now()" }

if (p.end_ts > now()) { p.end_ts = now(); log.warn('BACKTESTER_WINDOW_CLAMPED') }

{ "strategy": "sum_to_one_arb" }

if (!strategyRegistry.has(p.strategy)) throw ConfigError('BACKTESTER_UNKNOWN_STRATEGY')

{ "parameter_sweep": [] }

if (p.parameter_sweep.length > p.hard) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL')

8. Default Configuration

{
  "bot_id": "gov.backtester",
  "version": "2.0.0",
  "mode": "replay",
  "defaults": {
    "start_ts": "now()-7d",
    "end_ts": "now()",
    "strategy": "sum_to_one_arb",
    "parameter_sweep": []
  },
  "locked": {
    "mode": {
      "immutable": true,
      "value": "replay"
    }
  }
}

9. Implementation Flow

1. On run start, validate start_ts < end_ts and strategy is in the registry; reject with BACKTESTER_INVALID_WINDOW or BACKTESTER_UNKNOWN_STRATEGY otherwise.
2. Assign a replay_run_id (ULID) for this run; all emitted reports for this run carry replay_run_id and mode=replay.
3. Fetch archived ObservationReport, DecisionReport, RiskVote, ExecutionReport, and SettlementReport envelopes for the window [start_ts, end_ts] from the report archive.
4. For each tick in the replay window (tick = snapshot boundary from historical CLOB snapshots), reconstruct the order book state and available signals at that moment.
5. Feed reconstructed state and archived signals into the strategy under test using the same execution path as live (same guardrails, same execution logic).
6. For each simulated OrderIntent, run the full risk guardrail stack (replay mode — votes are simulated, not live); record each vote outcome.
7. Simulate fill outcomes using the historical order book snapshot at the matching tick; compute simulated fill price, slippage, and fee using pUSD denomination.
8. Emit replay-tagged OperationsReport after each simulated fill: includes replay_run_id, original_trace_id, simulated verdict, simulated P&L in pUSD, and comparison against original outcome.
9. Emit replay-tagged copies of simulated DecisionReport and ExecutionReport envelopes with mode=replay and replay_run_id for archive and audit.
10. At run completion, emit an aggregate OperationsReport summarising: total ticks replayed, simulated fills, simulated P&L, parameter variant results if sweep was configured, and comparison deltas vs original outcomes.
11. Retain all replay-tagged reports in polytraders.reports.operations for 1 year, partitioned by bot_slug+epoch.

10. Reference Implementation

Loads historical CLOB snapshots and archived report envelopes for the replay window, reconstructs the execution path tick by tick, emits replay-tagged OperationsReport per simulated fill and an aggregate report at run end. All outputs carry mode=replay and replay_run_id.

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. Translate to TS/Python/Go/Rust.

// ---- STARTUP ----
FUNCTION initReplay(config):
  IF config.start_ts >= config.end_ts:
    RAISE ConfigError('BACKTESTER_INVALID_WINDOW')
  IF NOT strategyRegistry.has(config.strategy):
    RAISE ConfigError('BACKTESTER_UNKNOWN_STRATEGY')
  IF config.end_ts > now():
    config.end_ts = now()
    alerting.emit('BACKTESTER_WINDOW_CLAMPED')
  replay_run_id = generateULID()  // e.g. 'replay_01HX9KZQ7E8VR5'
  variants = config.parameter_sweep OR [config.strategy.defaults]

// ---- PER-VARIANT LOOP ----
FOR variant IN variants:
  strategy = strategyRegistry.load(config.strategy, params=variant)

  // Fetch archived signals for the full window
  archive = FETCH internal.reportArchive.GET({
    window:   [config.start_ts, config.end_ts],
    kinds:    ['ObservationReport', 'DecisionReport', 'RiskVote',
               'ExecutionReport', 'SettlementReport']
  })

  // Fetch archived CLOB snapshots (tick-level)
  snapshots = FETCH internal.clobArchive.GET({
    window:   [config.start_ts, config.end_ts],
    resolution: 'tick'
  })

  sim_fills = []
  sim_pnl_pusd = 0.0

  FOR tick IN snapshots:
    // Reconstruct order book at this tick
    book = reconstructBook(tick)

    // Replay archived signals at this tick timestamp
    signals = archive.signalsAt(tick.ts_ms)

    // Run strategy in replay mode
    intent = strategy.evaluate(book, signals, mode='replay')
    IF intent IS NULL:
      CONTINUE

    // Simulate risk guardrail stack (replay mode)
    votes = riskStack.simulate(intent, context={
      book: book, signals: signals, mode: 'replay'
    })
    shaped_intent = applyVotes(intent, votes)
    IF shaped_intent IS NULL:
      EMIT OperationsReport(event_type='BACKTESTER_INTENT_REJECTED',
                            replay_run_id=replay_run_id, mode='replay', ...)
      CONTINUE

    // Simulate fill against historical book
    sim_fill = simulateFill(shaped_intent, book)
    sim_fill.size_pusd   = toPusdUnits(sim_fill.size_usd)
    sim_fill.fee_pusd    = sim_fill.size_pusd * sim_fill.fee_bps / 10_000
    sim_fill.pnl_pusd    = computeRealisedPnL(sim_fill, costBasis='FIFO')
    sim_fills.append(sim_fill)
    sim_pnl_pusd += sim_fill.pnl_pusd

    // Emit per-fill replay OperationsReport
    EMIT OperationsReport({
      report_id:       'ops_backtester_' + replay_run_id + '_' + tick.ts_ms,
      bot_id:          'gov.backtester',
      event_type:      'BACKTESTER_TICK_PROCESSED',
      replay_run_id:   replay_run_id,
      mode:            'replay',
      tick_ts_ms:      tick.ts_ms,
      market_id:       shaped_intent.market_id,
      simulated_fill:  sim_fill,
      risk_votes:      votes,
      topic:           'polytraders.reports.operations',
      partition:       'backtester+' + epochBucket(tick.ts_ms)
    })

  // Aggregate summary report at run end
  EMIT OperationsReport({
    event_type:              'BACKTESTER_REPLAY_COMPLETE',
    replay_run_id:           replay_run_id,
    strategy:                config.strategy,
    parameter_variant:       variant,
    mode:                    'replay',
    window_start:            config.start_ts,
    window_end:              config.end_ts,
    ticks_replayed:          len(snapshots),
    simulated_fills:         len(sim_fills),
    simulated_volume_pusd:   SUM(f.size_pusd FOR f IN sim_fills),
    simulated_pnl_pusd:      sim_pnl_pusd,
    simulated_net_fees_pusd: SUM(f.fee_pusd FOR f IN sim_fills),
    topic:                   'polytraders.reports.operations',
    partition:               'backtester+' + epochBucket(config.end_ts),
    retained_until:          config.end_ts + 1y
  })

SDK calls used

• internal.reportArchive.GET({ window, kinds })
• internal.clobArchive.GET({ window, resolution: 'tick' })
• strategyRegistry.load(slug, params)
• riskStack.simulate(intent, context)
• toPusdUnits(raw_usd)
• alerting.emit('BACKTESTER_WINDOW_CLAMPED', metadata)

Complexity: O(T * V) where T = ticks in replay window, V = parameter_sweep variants

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Backtester run configuration",
  "source": "config_store",
  "payload": {
    "bot_id": "gov.backtester",
    "mode": "replay",
    "strategy": "sum_to_one_arb",
    "start_ts": "2026-05-01T00:00:00Z",
    "end_ts": "2026-05-08T00:00:00Z",
    "parameter_sweep": [
      {
        "min_edge_bps": 12,
        "max_size_pusd": 1000
      },
      {
        "min_edge_bps": 15,
        "max_size_pusd": 1500
      }
    ]
  }
}

Output — what the bot emits

{
  "label": "OperationsReport — BACKTESTER_REPLAY_COMPLETE (2-variant sweep)",
  "payload": {
    "report_id": "ops_backtester_replay_01HX9KZQ7E8VR5",
    "bot_id": "gov.backtester",
    "event_type": "BACKTESTER_REPLAY_COMPLETE",
    "replay_run_id": "replay_01HX9KZQ7E8VR5",
    "strategy": "sum_to_one_arb",
    "mode": "replay",
    "window_start": "2026-05-01T00:00:00Z",
    "window_end": "2026-05-08T00:00:00Z",
    "ticks_replayed": 483120,
    "parameter_variants_run": 2,
    "variants": [
      {
        "variant_id": 0,
        "params": {
          "min_edge_bps": 12,
          "max_size_pusd": 1000
        },
        "simulated_fills": 214,
        "simulated_volume_pusd": 92450.0,
        "simulated_pnl_pusd": 1380.5,
        "simulated_net_fees_pusd": 231.1
      },
      {
        "variant_id": 1,
        "params": {
          "min_edge_bps": 15,
          "max_size_pusd": 1500
        },
        "simulated_fills": 178,
        "simulated_volume_pusd": 104200.0,
        "simulated_pnl_pusd": 1890.2,
        "simulated_net_fees_pusd": 260.5
      }
    ],
    "report_kind": "OperationsReport",
    "topic": "polytraders.reports.operations",
    "partition": "backtester+2026-05-08T00:00Z",
    "retained_until": "2027-05-08"
  }
}

12. Decision Logic

13. Standard Decision Output

This bot returns a OperationsReport object. See OperationsReport schema.

{
  "report_id": "ops_backtester_replay_01HX9KZQ7E8VR5",
  "bot_id": "gov.backtester",
  "event_type": "BACKTESTER_REPLAY_COMPLETE",
  "replay_run_id": "replay_01HX9KZQ7E8VR5",
  "strategy": "sum_to_one_arb",
  "mode": "replay",
  "window_start": "2026-05-01T00:00:00Z",
  "window_end": "2026-05-08T00:00:00Z",
  "ticks_replayed": 483120,
  "simulated_fills": 214,
  "simulated_volume_pusd": 92450.0,
  "simulated_pnl_pusd": 1380.5,
  "simulated_net_fees_pusd": 231.1,
  "vs_original_pnl_delta_pusd": 42.0,
  "parameter_variants_run": 1,
  "report_kind": "OperationsReport",
  "topic": "polytraders.reports.operations",
  "partition": "backtester+2026-05-08T00:00Z",
  "retained_until": "2027-05-08"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

Dashboards

• Grafana — Governance / Backtester run history and simulated P&L by strategy
• Grafana — Governance / Parameter sweep results comparison

16. Developer Reporting

{
  "bot_id": "gov.backtester",
  "event_type": "BACKTESTER_TICK_PROCESSED",
  "replay_run_id": "replay_01HX9KZQ7E8VR5",
  "tick_ts_ms": 1746792060000,
  "market_id": "0x9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c",
  "simulated_intent_emitted": true,
  "simulated_fill_price": 0.621,
  "simulated_fill_size_pusd": 430.0,
  "simulated_slippage_bps": 0.8,
  "risk_votes": [
    {
      "bot": "liquidityguard",
      "vote": "pass"
    },
    {
      "bot": "portfolioguard",
      "vote": "reshape",
      "new_size_pusd": 430.0
    }
  ],
  "mode": "replay"
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

On restart, any in-progress replay run is marked as interrupted. A new run must be launched manually. No partial replay state is held in memory.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

Sibling bots (same OrderIntent)

Used by (auto-aggregated)

6.8

External services

23. Security Surfaces

Abuse vectors considered

• Submitting a large parameter_sweep (>200 variants) to exhaust archive query budget
• Injecting a crafted replay window that targets gaps in the archive to produce misleading P&L results
• Manipulating strategy registry to replay a strategy not approved for backtesting

Mitigations

• parameter_sweep hard limit of 200 enforced at config load; excess rejected with PARAMETER_CHANGE_REQUIRES_APPROVAL
• Backtester is locked to mode=replay; live CLOB and onchain surfaces are never accessed
• Strategy registry is read-only within the backtester; only registered strategies can be loaded
• All replay runs are audit-logged with replay_run_id and emitted to the governance report bus

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

Backtester incidents involve archive unavailability (run aborts), high tick-skip rates (data quality issue), or runs stuck in progress (goroutine leak or resource exhaustion). All incidents are low-urgency unless they block a promotion gate decision.

On-call actions

Manual overrides

• polytraders gov backtest abort --replay-run-id <id> — Aborts a stuck replay run; use after investigating root cause. Originally: A replay run is stuck in progress and blocking the run queue.

Healthcheck

/internal/health/backtester → 200 Last run completed < 24h ago; archive reachable; max_in_flight not saturated; ticks_skipped_rate < 0.01.; red if Archive unreachable; all runs aborting; run queue saturated for > 5 minutes.

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%