1.12 ModelDriftMonitor

1. Bot Identity

Operational profile

2. Purpose

ModelDriftMonitor flags strategies whose live behaviour has decoupled from their backtest distribution. It computes a drift score by comparing the rolling distribution of live signals or fill prices against the expected backtest baseline and hard-rejects new orders when drift exceeds the configured ceiling, preventing a degraded model from continuing to place orders.

3. Why This Bot Matters

• Model degradation undetected

  A strategy whose signal quality has degraded continues placing orders, accumulating losses without any automated circuit-breaker.

• Market regime change

  A model trained on one market regime may produce pathological signals in a new regime; model drift detection provides an early-warning gate before significant capital is deployed.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "max_drift_score": 0.25 }

if (driftScore > params.max_drift_score) return reject('MODEL_DRIFT_EXCEEDED');

{ "drift_lookback_n": 50 }

if (liveObs.length < params.drift_lookback_n) return approve('MODEL_DRIFT_SKIPPED');

{ "drift_metric": "ks_statistic" }

const driftScore = computeDrift(liveObs, backtestBaseline, params.drift_metric);

8. Default Configuration

{
  "bot_id": "risk.model_drift_monitor",
  "version": "0.1.0",
  "mode": "hard_guard",
  "defaults": {
    "max_drift_score": 0.25,
    "drift_lookback_n": 50,
    "drift_metric": "ks_statistic"
  },
  "locked": {
    "max_drift_score": {
      "max": 0.5
    }
  }
}

9. Implementation Flow

1. Receive OrderIntent from a model-driven strategy with strategy_id.
2. Check KillSwitch; if active, HARD_REJECT(KILL_SWITCH_ACTIVE).
3. Load backtest baseline distribution for strategy_id from internal store.
4. If baseline unavailable, HARD_REJECT(MODEL_DRIFT_DATA_UNAVAILABLE).
5. Fetch recent fill prices and signal values for strategy_id.
6. If fewer than drift_lookback_n observations, APPROVE (skip — insufficient data).
7. Compute drift_score using configured drift_metric (KS or PSI).
8. If drift_score > max_drift_score, HARD_REJECT(MODEL_DRIFT_EXCEEDED).
9. If drift_score > warning threshold, attach WARN annotation; APPROVE.
10. APPROVE with drift_score attached to the RiskVote.

10. Reference Implementation

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. IF/THEN/ELSE = decision. Translate directly to TypeScript, Python, Go, or Rust.

FUNCTION evaluateModelDrift(intent):
  ks = FETCH internal.killswitch.status
  IF ks.active: EMIT RiskVote(HARD_REJECT, KILL_SWITCH_ACTIVE); RETURN

  baseline = FETCH internal.backtest_baseline(intent.strategy_id)
  IF baseline IS NULL:
    EMIT RiskVote(HARD_REJECT, MODEL_DRIFT_DATA_UNAVAILABLE); RETURN

  liveObs = FETCH clob_auth.fill_prices(intent.strategy_id, n=params.drift_lookback_n)
  IF len(liveObs) < params.drift_lookback_n:
    EMIT RiskVote(APPROVE); RETURN  // insufficient data — skip

  driftScore = compute_ks(liveObs, baseline.distribution)

  IF driftScore > params.max_drift_score:
    EMIT RiskVote(HARD_REJECT, MODEL_DRIFT_EXCEEDED,
                  drift_score=driftScore); RETURN
  IF driftScore > params.max_drift_score * 0.6:
    annotations.append(WARN(MODEL_DRIFT_WARN, drift_score=driftScore))

  EMIT RiskVote(APPROVE, drift_score=driftScore)

SDK calls used

• clob_auth.fill_prices(strategy_id, n)
• internal.backtest_baseline(strategy_id)
• internal.killswitch.status()

Complexity: O(N log N) for KS statistic where N = drift_lookback_n (max 50)

11. Wire Examples

Input — what arrives on the wire

OrderIntent — drift exceeded — internal

{
  "intent_id": "int_d4e5f6a7b8c90004",
  "strategy_id": "strat_002",
  "size_usd": 200,
  "generated_at_ms": 1746800000000
}

Output — what the bot emits

RiskVote — HARD_REJECT

{
  "guard_id": "risk.model_drift_monitor",
  "decision": "HARD_REJECT",
  "severity": "HARD",
  "reason_code": "MODEL_DRIFT_EXCEEDED",
  "message": "Drift score 0.32 exceeds ceiling 0.25 for strategy strat_002.",
  "constraints": {},
  "checked_at": "2026-05-10T11:00:00Z"
}

12. Decision Logic

13. Standard Decision Output

This bot returns a RiskVote object. See RiskVote schema.

{
  "guard_id": "risk.model_drift_monitor",
  "decision": "HARD_REJECT",
  "severity": "HARD",
  "reason_code": "MODEL_DRIFT_EXCEEDED",
  "message": "Strategy strat_002 drift score 0.32 exceeds ceiling 0.25. New orders blocked.",
  "constraints": {},
  "inputs_used": [
    "internal.backtest_baseline",
    "clob_auth.fills"
  ],
  "checked_at": "2026-05-10T11:00:00Z"
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

16. Developer Reporting

{
  "bot_id": "risk.model_drift_monitor",
  "decision": "HARD_REJECT",
  "reason_code": "MODEL_DRIFT_EXCEEDED",
  "inputs_used": [
    "internal.backtest_baseline",
    "clob_auth.fills"
  ],
  "metrics": {
    "strategy_id": "strat_002",
    "drift_score": 0.32,
    "drift_metric": "ks_statistic",
    "lookback_n": 50,
    "ceiling": 0.25
  },
  "checked_at": "2026-05-10T11:00:00Z"
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

Baseline reloaded from Redis on cold start. If unavailable, HARD_REJECT until restored.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

External services

23. Security Surfaces

Abuse vectors considered

• Injecting synthetic fill data to deflate the drift score
• Bypassing drift check by submitting from a strategy_id with no backtest baseline

Mitigations

• Fill data sourced exclusively from CLOB authenticated endpoint with provenance timestamp
• Missing baseline triggers HARD_REJECT (fail-closed); unknown strategy_id is treated as missing baseline

24. Polymarket V2 Compatibility

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

ModelDriftMonitor incidents typically indicate a genuine strategy degradation or a regime change. Verify whether the drift is expected before unlocking the strategy.

On-call actions

Manual overrides

• polytraders risk update-baseline --strategy-id <id> --source recent_fills — After a confirmed intentional strategy update that changes the expected fill distribution.

Healthcheck

GET /internal/health/modeldriftmonitor → green: Baselines loaded for all active strategies, fill history accessible, p99 eval latency < 150ms; red: Any strategy baseline missing, fill history unavailable, or HARD_REJECT rate > 0.1

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%