6.9 ExperimentTracker

1. Bot Identity

2. Purpose

ExperimentTracker manages shadow and limited-live A/B experiments, records matched-pair samples, computes confidence intervals, and emits a drift signal to StrategyRegistry when a variant underperforms.

3. Why This Bot Matters

• No experiment tracking

  Promotions are made without statistical evidence; regressions go undetected.

• Auto-promote without human sign-off

  A variant with a transient winning streak is promoted before significance is established.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "min_samples_for_decision": 100 }

if samples < p.min_samples_for_decision: emit('EXPERIMENT_INSUFFICIENT_SAMPLES')

{ "traffic_split_pct": 10 }

if p.traffic_split_pct > 50: emit('EXPERIMENT_LARGE_SPLIT_WARN')

8. Default Configuration

{
  "bot_id": "gov.experimenttracker",
  "version": "0.1.0",
  "mode": "shadow_only",
  "defaults": {
    "min_samples_for_decision": 100,
    "traffic_split_pct": 10,
    "auto_promote_on_winning": false,
    "require_human_signoff": true
  }
}

9. Implementation Flow

1. On experiment start, assign variant_id and record traffic_split_pct and baseline strategy slug.
2. For each matched pair (shadow fill vs live fill), record edge, slippage, and fill quality in pUSD.
3. Compute running confidence intervals on edge delta between variant and control.
4. When samples >= min_samples_for_decision and CI is significant, emit EXPERIMENT_RESULT report.
5. If variant underperforms control by > 2 sigma, emit drift signal to StrategyRegistry.
6. If require_human_signoff=true, block auto-promote even when variant wins.

10. Reference Implementation

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. IF/THEN/ELSE = decision. Translate directly to TypeScript, Python, Go, or Rust.

// ---- EXPERIMENT START ----
FUNCTION startExperiment(config):
  exp = {id: generateULID(), variant: config.variant_slug,
         control: config.control_slug, samples: [], started_at: now()}
  postgres.insert('experiments', exp)
  EMIT OperationsReport(event_type='EXPERIMENT_STARTED', experiment_id=exp.id)

// ---- SAMPLE RECORDING ----
FUNCTION recordSample(variantFill, controlFill, experimentId):
  delta_bps = (variantFill.edge_pusd - controlFill.edge_pusd) / controlFill.notional * 10000
  postgres.insert('experiment_samples', {experiment_id: experimentId,
    variant_fill_pusd: variantFill.size_pusd,
    control_fill_pusd: controlFill.size_pusd,
    edge_delta_bps: delta_bps, recorded_at: now()})

// ---- RESULT EVALUATION ----
FUNCTION evaluateExperiment(experimentId):
  samples = postgres.select('experiment_samples', WHERE experiment_id=experimentId)
  IF len(samples) < config.min_samples_for_decision:
    EMIT OperationsReport(event_type='EXPERIMENT_INSUFFICIENT_SAMPLES')
    RETURN
  ci = computeCI95(samples)
  verdict = 'variant_wins' IF ci.low > 0 ELSE 'control_wins' IF ci.high < 0 ELSE 'inconclusive'
  EMIT OperationsReport(event_type='EXPERIMENT_RESULT', verdict=verdict,
    ci_95_low=ci.low, ci_95_high=ci.high)
  IF verdict == 'control_wins':
    strategyRegistry.sendDriftSignal(experimentId.variant_slug)

SDK calls used

• postgres.insert('experiments', exp)
• postgres.select('experiment_samples', ...)
• strategyRegistry.sendDriftSignal(slug)

Complexity: O(S) per evaluation where S = sample count

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Matched-pair sample",
  "source": "internal.report_bus",
  "payload": {
    "experiment_id": "exp_sports_v2",
    "variant_fill_pusd": 430.0,
    "control_fill_pusd": 415.0,
    "recorded_at_ms": 1746792060000
  }
}

Output — what the bot emits

{
  "label": "OperationsReport — EXPERIMENT_RESULT",
  "payload": {
    "report_id": "ops_exp_01HX9Z",
    "event_type": "EXPERIMENT_RESULT",
    "verdict": "variant_wins",
    "ci_95_low": 1.1,
    "ci_95_high": 5.3,
    "report_kind": "OperationsReport",
    "topic": "polytraders.reports.operations"
  }
}

12. Decision Logic

13. Standard Decision Output

This bot returns a OperationsReport object. See OperationsReport schema.

{
  "report_id": "ops_experimenttracker_01HX9Z",
  "bot_id": "gov.experimenttracker",
  "event_type": "EXPERIMENT_RESULT",
  "experiment_id": "exp_sports_v2",
  "variant_slug": "sports-model-v2",
  "control_slug": "sports-model",
  "samples": 150,
  "edge_delta_bps": 3.2,
  "ci_95_low": 1.1,
  "ci_95_high": 5.3,
  "verdict": "variant_wins",
  "report_kind": "OperationsReport",
  "topic": "polytraders.reports.operations"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

16. Developer Reporting

{
  "bot_id": "gov.experimenttracker",
  "event_type": "SAMPLE_RECORDED",
  "experiment_id": "exp_sports_v2",
  "sample_n": 47,
  "variant_fill_pusd": 430.0,
  "control_fill_pusd": 415.0,
  "edge_delta_bps": 3.6
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

On restart, reload active experiments from Postgres; resume sampling from last recorded sample.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

Sibling bots (same OrderIntent)

External services

23. Security Surfaces

Abuse vectors considered

• Manipulating sample data to bias experiment toward a preferred variant

Mitigations

• Samples are immutably written to Postgres; no update path exists on experiment_samples

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

ExperimentTracker incidents involve stalled sampling (bus unavailable) or drift signals blocking a planned promotion.

On-call actions

Manual overrides

• —

Healthcheck

/internal/health/experimenttracker → green if Postgres reachable; at least one active experiment has received samples in the last hour; red if No samples recorded in 2h for any active experiment

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%