2.7 FillQualityAnalyzer

1. Bot Identity

Operational profile

2. Purpose

FillQualityAnalyzer scores every completed fill on price slippage, execution latency, and post-fill toxicity relative to the original intent. Low-scoring fills generate WARN annotations; systematic poor scores feed back to strategy tuning.

3. Why This Bot Matters

• Slippage not tracked per fill

  Execution costs compound silently; strategy returns eroded without visibility into the cause.

• Post-fill toxicity not measured

  The system continues submitting orders on markets with chronic adverse selection, accumulating losses.

• Fill latency not profiled

  Slow fills on fast-moving markets indicate queue-position issues that smarter order type selection could fix.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "toxicity_horizon_s": 30 }

assert params.toxicity_horizon_s <= params.hard

{ "min_score_for_repeat": 0.6 }

if score < params.min_score_for_repeat: emit(FILL_QUALITY_BELOW_THRESHOLD)

{ "warn_threshold_bps": 20 }

if slippage_bps > params.warn_threshold_bps: emit(FILL_SLIPPAGE_ELEVATED)

{ "publish_to": ["polytraders.reports.execution"] }

assert len(params.publish_to) > 0

8. Default Configuration

{
  "bot_id": "exec.fillqualityanalyzer",
  "version": "0.1.0",
  "mode": "shadow_only",
  "defaults": {
    "toxicity_horizon_s": 30,
    "min_score_for_repeat": 0.6,
    "warn_threshold_bps": 20,
    "publish_to": [
      "polytraders.reports.execution"
    ]
  },
  "locked": {
    "toxicity_horizon_s": {
      "max": 120
    },
    "warn_threshold_bps": {
      "max": 100
    }
  }
}

9. Implementation Flow

1. Receive ExecutionReport(FILLED) from OrderLifecycleManager.
2. Fetch CLOB V2 top-of-book snapshot from clob_public at fill_ts_ms to get arrival_mid.
3. Compute slippage_bps = abs(fill_price - arrival_mid) / arrival_mid * 10000.
4. Compute latency_ms = fill_ts_ms - submit_ms from the ExecutionReport.
5. Wait toxicity_horizon_s; fetch trade tape from ws_market to compute post-fill drift_bps.
6. Score dimensions: price_score = max(0, 1 - slippage_bps/100); latency_score = max(0, 1 - latency_ms/500); toxicity_score = max(0, 1 - drift_bps/100).
7. Overall score = mean(price_score, latency_score, toxicity_score).
8. Annotate ExecutionReport with scores and reason codes (FILL_QUALITY_BELOW_THRESHOLD if score < min_score_for_repeat).
9. Emit annotated ExecutionReport to publish_to topics.

10. Reference Implementation

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. IF/THEN/ELSE = decision. Translate directly to TypeScript, Python, Go, or Rust.

FUNCTION analyzeFill(executionReport):
  // 1. Fetch arrival mid at fill time
  book = FETCH clob_public.GET('/book?market=' + executionReport.market_id)
  arrivalMid = (book.best_bid + book.best_ask) / 2

  // 2. Slippage score
  slippageBps = abs(executionReport.fill_price - arrivalMid) / arrivalMid * 10000
  priceScore = max(0, 1 - slippageBps / 100)
  IF slippageBps > params.warn_threshold_bps:
    EMIT reason(FILL_SLIPPAGE_ELEVATED)

  // 3. Latency score
  latencyMs = executionReport.fill_ts_ms - executionReport.submit_ms
  latencyScore = max(0, 1 - latencyMs / 500)

  // 4. Toxicity score (deferred by toxicity_horizon_s)
  WAIT params.toxicity_horizon_s * 1000
  tape = FETCH ws_market.recent_fills(executionReport.market_id, last_n=100)
  IF tape IS NULL:
    toxicityScore = None
    EMIT reason(FILL_QUALITY_TOXICITY_UNAVAILABLE, WARN)
  ELSE:
    driftBps = computeDrift(tape, executionReport.fill_price, params.toxicity_horizon_s)
    toxicityScore = max(0, 1 - driftBps / 100)

  // 5. Overall score and verdict
  scores = [s FOR s IN [priceScore, latencyScore, toxicityScore] IF s IS NOT None]
  overallScore = mean(scores)
  IF overallScore < 0.2:
    verdict = 'FILL_QUALITY_CRITICAL'
  ELIF overallScore < params.min_score_for_repeat:
    verdict = 'FILL_QUALITY_BELOW_THRESHOLD'
  ELSE:
    verdict = 'FILL_QUALITY_PASS'

  // 6. Annotate and emit
  executionReport.fill_quality = {slippageBps, latencyMs, driftBps, overallScore, verdict}
  EMIT ExecutionReport(executionReport) TO params.publish_to

SDK calls used

• clob_public.GET('/book?market=' + market_id)
• ws_market.recent_fills(market_id, last_n=100)

Complexity: O(F) where F = tape size (capped at 100)

11. Wire Examples

Input — what arrives on the wire

ExecutionReport (FILLED) from OrderLifecycleManager — exec.orderlifecyclemanager

{
  "order_id": "0xbbbb2222cccc3333dddd4444eeee5555ffff6666aaaa7777bbbb8888cccc9999",
  "status": "FILLED",
  "fill_price": 0.621,
  "fill_usd": 450,
  "submit_ms": 1746770000000,
  "fill_ts_ms": 1746770000085,
  "collateral": "pUSD"
}

Output — what the bot emits

Annotated ExecutionReport with fill quality scores

{
  "report_id": "rep_2b3c4d5e6f7a8b9c",
  "bot_id": "exec.fillqualityanalyzer",
  "verdict": "FILL_QUALITY_PASS",
  "slippage_bps": 1.6,
  "latency_ms": 85,
  "drift_bps": 4,
  "overall_score": 0.92,
  "collateral": "pUSD",
  "evaluated_at_ms": 1746770030000
}

12. Decision Logic

13. Standard Decision Output

This bot returns a ExecutionReport object. See ExecutionReport schema.

{
  "report_id": "rep_2b3c4d5e6f7a8b9c",
  "trace_id": "trc_1a2b3c4d5e6f7a8b",
  "bot_id": "exec.fillqualityanalyzer",
  "order_id": "0xbbbb2222cccc3333dddd4444eeee5555ffff6666aaaa7777bbbb8888cccc9999",
  "market_id": "0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b",
  "fill_price": 0.621,
  "arrival_mid": 0.62,
  "slippage_bps": 1.6,
  "latency_ms": 85,
  "drift_bps": 4,
  "price_score": 0.98,
  "latency_score": 0.83,
  "toxicity_score": 0.96,
  "overall_score": 0.92,
  "verdict": "FILL_QUALITY_PASS",
  "collateral": "pUSD",
  "evaluated_at_ms": 1746770030000
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

16. Developer Reporting

{
  "report_id": "rep_2b3c4d5e6f7a8b9c",
  "slippage_bps": 1.6,
  "latency_ms": 85,
  "drift_bps": 4,
  "price_score": 0.98,
  "latency_score": 0.83,
  "toxicity_score": 0.96,
  "overall_score": 0.92,
  "min_score_for_repeat": 0.6,
  "warn_threshold_bps": 20
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

WAL replayed on restart; in-memory cache rebuilt from last 1000 fills.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

External services

23. Security Surfaces

Abuse vectors considered

• Injecting a fabricated FILLED event with a favourable fill_price to suppress a poor-quality fill from scoring
• Replaying old ExecutionReports to inflate fill quality metrics

Mitigations

• ExecutionReport messages validated against originating bot HMAC before scoring
• Idempotency key (order_id + fill_ts_ms) prevents duplicate scoring

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

FillQualityAnalyzer incidents are typically high critical fill rates (systematic adverse selection) or stale market tape causing toxicity measurement failures.

On-call actions

Manual overrides

• polytraders bot rescore exec.fillqualityanalyzer --order <order_id> — Rescore a fill after feed availability is restored to get accurate toxicity_score.

Healthcheck

GET /internal/health/fillqualityanalyzer → green if ws_market feed connected, clob_public reachable, critical_total rate < 0.05/min, pending toxicity evals < 100; red if ws_market disconnected, critical rate > 0.2/min, pending evals > 100

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%