Polytraders Dev Guide
internal
v3 spine Phase 1 · Shared contracts 9 demo-wired · 0 shadow-ready · 0 production-live · 100 pending · 109 total 15/33 infra tasks the plan status board
HomeBy LayerGovernance6.6 Cron Runner

6.6 Cron Runner

Governance Governance Service Explain LIVE General live capital · Indirect P7 · Governance & replay pending stub

CronRunner maintains the authoritative cron schedule for the entire Polytraders system. It reads a declarative cron manifest (cron_expression per task, UTC), fires each task at the correct wall-clock moment, serialises concurrent firings into an ordered queue, and emits an OperationsReport heartbeat after each dispatch. CronRunner never executes strategy logic or touches the CLOB directly; it is purely a scheduling and dispatch service. Internal-only — no API surface beyond the internal bus.

v3 readiness

Docs27/27
donehow scored
Impl0/15
pendinghow scored
Backtest0/4
pendinghow scored
Runtime0/8
pendinghow scored

A bot is done when all four scores are. What does done mean?

1. Bot Identity

LayerGovernance  Governance
Bot classGovernance Service
AuthorityExplain
StatusLIVE
ReadinessGeneral live
Runs beforeEvery scheduled bot task — CronRunner fires the trigger
Runs afterSystem startup
Applies toAll cron-scheduled tasks across all 97 production bots
Default modegeneral_live
User-visibleAdvanced details only
Developer ownerPolytraders core — Governance pod

2. Purpose

CronRunner maintains the authoritative cron schedule for the entire Polytraders system. It reads a declarative cron manifest (cron_expression per task, UTC), fires each task at the correct wall-clock moment, serialises concurrent firings into an ordered queue, and emits an OperationsReport heartbeat after each dispatch. CronRunner never executes strategy logic or touches the CLOB directly; it is purely a scheduling and dispatch service. Internal-only — no API surface beyond the internal bus.

3. Why This Bot Matters

  • Cron task not fired on schedule

    Downstream bots that rely on scheduled triggers (e.g. daily P&L reconciliation, pre-event quoting cycles) silently miss their window. Delayed reconciliation compounds drift.

  • Two instances of CronRunner fire the same task simultaneously

    Duplicate fills or double-counted reconciliation records corrupt ledgers and inflate reported P&L.

  • CronRunner crashes and does not recover

    All scheduled tasks across all 97 bots stop firing. The system drifts into an uncontrolled state without time-based governance.

  • Missed quiet-hours suppression

    Strategies fire during configured quiet hours (e.g. market-open volatility windows), creating unintended exposure.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

InputSourceRequired?Use
Market schedule metadata (event start times)internalNoAlign pre-event quoting cycles to Polymarket event start times when calendar feeds are configured.

5. Required Internal Inputs

InputSourceRequired?Use
Cron manifest — list of tasks with cron_expression, enabled_strategies, quiet_hour windowsConfig storeYesPrimary schedule definition. Each entry maps a cron expression to one or more strategy or governance task IDs.
KillSwitch active flagKillSwitchNoWhen KillSwitch is active, suppress all non-critical scheduled tasks. Health and governance tasks continue.

6. Parameter Guide

ParameterDefaultWarningHardWhat it controls
cron_expression0 * * * *NoneNoneStandard POSIX cron expression (UTC) defining when a scheduled task fires.
enabled_strategies[]NoneNoneList of strategy or bot IDs that this cron entry should trigger.
disable_during_quiet_hoursFalseNoneNoneWhen true, suppresses this task during configured system quiet hours.
max_concurrent_tasks204060Maximum number of cron tasks that may be dispatched concurrently in a single tick.

7. Detailed Parameter Instructions

cron_expression

What it means

Standard POSIX cron expression (UTC) defining when a scheduled task fires.

Default

{ "cron_expression": "0 * * * *" }

Why this default matters

Hourly default prevents unintentional high-frequency dispatch. Each task must explicitly set its own expression.

Threshold logic

ConditionAction
valid cron expressionSchedule task
invalid expressionReject at config load; emit CRON_RUNNER_INVALID_EXPRESSION

Developer check

parseCron(p.cron_expression) || throw ConfigError('CRON_RUNNER_INVALID_EXPRESSION')

User-facing English

The system runs certain maintenance and reporting tasks on a fixed schedule.

enabled_strategies

What it means

List of strategy or bot IDs that this cron entry should trigger.

Default

{ "enabled_strategies": [] }

Why this default matters

Empty list by default — tasks must be explicitly enabled to prevent accidental firing of inactive strategies.

Threshold logic

ConditionAction
non-empty listDispatch trigger to each listed bot ID
empty listLog warning CRON_RUNNER_NO_TARGETS and skip

Developer check

if (!p.enabled_strategies.length) log.warn('CRON_RUNNER_NO_TARGETS')

User-facing English

Scheduled tasks run only for active components.

disable_during_quiet_hours

What it means

When true, suppresses this task during configured system quiet hours.

Default

{ "disable_during_quiet_hours": false }

Why this default matters

Off by default so critical governance tasks (health checks, reconciliation) always fire. Enable only for trading-adjacent tasks.

Threshold logic

ConditionAction
disable_during_quiet_hours=true AND now in quiet windowSkip task; emit CRON_RUNNER_QUIET_HOURS_SKIP
disable_during_quiet_hours=falseFire regardless of quiet hours

Developer check

if (p.disable_during_quiet_hours && isQuietHour(now)) { log.info('CRON_RUNNER_QUIET_HOURS_SKIP'); return; }

User-facing English

Some automated tasks are paused during quiet periods.

max_concurrent_tasks

What it means

Maximum number of cron tasks that may be dispatched concurrently in a single tick.

Default

{ "max_concurrent_tasks": 20 }

Why this default matters

At 20 concurrent tasks, the internal bus stays within throughput limits. Above 40, dispatch latency degrades.

Threshold logic

ConditionAction
max_concurrent_tasks <= 20Normal dispatch
20–60WARN — bus throughput may degrade
> 60Reject config change — PARAMETER_CHANGE_REQUIRES_APPROVAL

Developer check

if (p.max_concurrent_tasks > p.hard) throw ConfigError("PARAMETER_CHANGE_REQUIRES_APPROVAL")

User-facing English

— not yet authored —

8. Default Configuration

{
  "bot_id": "gov.cron_runner",
  "version": "2.0.0",
  "mode": "general_live",
  "defaults": {
    "cron_expression": "0 * * * *",
    "enabled_strategies": [],
    "disable_during_quiet_hours": false,
    "max_concurrent_tasks": 20
  }
}

9. Implementation Flow

  1. On startup, load the cron manifest from the config store; parse each entry's cron_expression (UTC) and validate.
  2. Register a ticker per unique cron expression; at each tick, collect all tasks whose expression matches the current UTC time.
  3. Before dispatching any task, check KillSwitch active flag; if active, suppress trading-adjacent tasks but allow health and governance tasks.
  4. For each eligible task, check disable_during_quiet_hours against the current UTC time window.
  5. Dispatch the task trigger to each enabled_strategies entry via the internal message bus; log dispatch with a trace_id.
  6. After dispatch, emit an OperationsReport heartbeat: { task_id, fired_at_ms, targets, suppressed_count, trace_id }.
  7. Detect and deduplicate concurrent firings using a distributed lock keyed on (task_id, scheduled_at_ms); if lock is held, skip and emit CRON_RUNNER_DUPLICATE_FIRE.
  8. Retain dispatch logs for 1 year for operational audit.

10. Reference Implementation

Loads a cron manifest, registers tickers, dispatches task triggers at UTC boundaries, deduplicates via distributed lock, and emits an OperationsReport per dispatch.

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output. Translate to TS/Python/Go/Rust.

// ---- STARTUP ----
FUNCTION init():
  manifest = FETCH config_store.GET('/cron-manifest')
  FOR entry IN manifest:
    validateCronExpression(entry.cron_expression)
    registerTicker(entry)  // fires onTick() when expression matches UTC now

// ---- PER-TICK HANDLER ----
FUNCTION onTick(entry, scheduled_at_ms):
  lock_key = entry.task_id + ':' + scheduled_at_ms
  IF NOT redis.SET(lock_key, NX=true, PX=5000):  // 5s TTL
    log.warn('CRON_RUNNER_DUPLICATE_FIRE', { task_id: entry.task_id })
    RETURN

  ks = FETCH killswitch.status()
  IF ks.active AND entry.task_class == 'trading':
    log.info('CRON_RUNNER_KILL_SWITCH_ACTIVE', { task_id: entry.task_id })
    RETURN

  IF entry.disable_during_quiet_hours AND isQuietHour(now()):
    log.info('CRON_RUNNER_QUIET_HOURS_SKIP', { task_id: entry.task_id })
    EMIT OperationsReport(event_type='CRON_TASK_SUPPRESSED', ...)
    RETURN

  IF NOT entry.enabled_strategies:
    log.warn('CRON_RUNNER_NO_TARGETS', { task_id: entry.task_id })
    RETURN

  trace_id = generateTraceId()
  FOR target IN entry.enabled_strategies:
    internal_bus.publish(topic='cron.trigger', payload={
      task_id:        entry.task_id,
      target:         target,
      scheduled_at:   scheduled_at_ms,
      trace_id:       trace_id
    })

  EMIT OperationsReport({
    report_id:        'ops_cronrunner_' + scheduled_at_ms,
    bot_id:           'gov.cron_runner',
    event_type:       'CRON_TASK_DISPATCHED',
    task_id:          entry.task_id,
    fired_at_ms:      scheduled_at_ms,
    targets:          entry.enabled_strategies,
    suppressed_count: 0,
    trace_id:         trace_id
  })

// ---- MANIFEST RELOAD ----
FUNCTION reloadManifest():
  // Called on config change event; validates and re-registers all tickers
  FOR entry IN FETCH config_store.GET('/cron-manifest'):
    validateCronExpression(entry.cron_expression)
  registerTickers(manifest)  // replaces existing registrations atomically

SDK calls used

  • config_store.GET('/cron-manifest')
  • redis.SET(lock_key, NX=true, PX=5000)
  • killswitch.status()
  • internal_bus.publish(topic='cron.trigger', payload=...)
  • alerting.emit('CRON_RUNNER_DUPLICATE_FIRE', metadata)

Complexity: O(T) per tick where T = tasks matching the current cron boundary

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Cron manifest entry",
  "source": "config_store",
  "payload": {
    "task_id": "daily_pnl_reconcile",
    "cron_expression": "0 0 * * *",
    "enabled_strategies": [
      "gov.pnl_reporter"
    ],
    "disable_during_quiet_hours": false,
    "task_class": "governance"
  }
}

Output — what the bot emits

{
  "label": "OperationsReport — CRON_TASK_DISPATCHED",
  "payload": {
    "report_id": "ops_cronrunner_1746792000000",
    "bot_id": "gov.cron_runner",
    "event_type": "CRON_TASK_DISPATCHED",
    "task_id": "daily_pnl_reconcile",
    "fired_at_ms": 1746792000000,
    "targets": [
      "gov.pnl_reporter"
    ],
    "suppressed_count": 0,
    "trace_id": "tr_abc123def456",
    "report_kind": "OperationsReport"
  }
}

12. Decision Logic

APPROVE

Not applicable — CronRunner dispatches triggers, it does not approve or reject trading decisions.

RESHAPE_REQUIRED

Not applicable.

REJECT

CronRunner will reject a task dispatch if KillSwitch is active for that task class, or if a duplicate lock is held.

WARNING_ONLY

Invalid cron expressions and empty target lists emit warnings but do not crash the scheduler.

13. Standard Decision Output

This bot returns a OperationsReport object. See OperationsReport schema.

{
  "report_id": "ops_cronrunner_20260509T120000Z",
  "bot_id": "gov.cron_runner",
  "event_type": "CRON_TASK_DISPATCHED",
  "task_id": "daily_pnl_reconcile",
  "fired_at_ms": 1746792000000,
  "targets": [
    "gov.pnl_reporter"
  ],
  "suppressed_count": 0,
  "trace_id": "tr_abc123def456",
  "report_kind": "OperationsReport"
}

14. Reason Codes

CodeSeverityMeaningActionUser-facing message
CRON_RUNNER_TASK_DISPATCHEDINFOA cron task fired on schedule and was dispatched to all configured targets.No action — operational heartbeat.
CRON_RUNNER_INVALID_EXPRESSIONHARD_REJECTA cron expression in the manifest failed to parse.Reject the config entry at load time; alert on-call.
CRON_RUNNER_DUPLICATE_FIREWARNA second CronRunner instance attempted to dispatch the same task at the same scheduled time; distributed lock was already held.Skip dispatch; log warning.
CRON_RUNNER_QUIET_HOURS_SKIPINFOA task with disable_during_quiet_hours=true was suppressed because the current UTC time falls within a configured quiet window.No action — expected behaviour.A scheduled task was paused during a quiet period.
CRON_RUNNER_NO_TARGETSWARNA task entry has an empty enabled_strategies list; there is nothing to dispatch.Log warning; skip dispatch.
KILL_SWITCH_ACTIVEWARNKillSwitch is active; trading-class scheduled tasks are suppressed.Suppress trading tasks; continue governance and health tasks.Scheduled trading tasks are paused while the system kill switch is active.
CRON_RUNNER_BUS_PUBLISH_FAILEDWARNInternal bus publish for a task trigger failed after retries.Log error; emit alert; retry on next tick.
CRON_RUNNER_MANIFEST_RELOAD_OKINFOCron manifest reloaded successfully after a config change event.No action.

15. Metrics & Logs

Metrics emitted

MetricTypeUnitLabelsMeaning
polytraders_gov_cronrunner_tasks_dispatched_totalcountercounttask_id, task_classTotal cron tasks dispatched successfully.
polytraders_gov_cronrunner_tasks_suppressed_totalcountercountreason, task_idTasks suppressed due to quiet hours, kill switch, or no targets.
polytraders_gov_cronrunner_duplicate_fire_totalcountercounttask_idDuplicate dispatch attempts blocked by distributed lock.
polytraders_gov_cronrunner_dispatch_latency_mshistogrammstask_idWall-clock latency from cron boundary to internal bus publish.
polytraders_gov_cronrunner_manifest_entriesgaugecountNumber of active entries in the loaded cron manifest.
polytraders_gov_cronrunner_bus_failures_totalcountercounttask_idInternal bus publish failures for cron triggers.

Alerts

AlertConditionSeverityRunbook
CronRunnerNoDispatchIn15mrate(polytraders_gov_cronrunner_tasks_dispatched_total[15m]) == 0page#runbook-cronrunner-no-dispatch
CronRunnerDuplicateFireRaterate(polytraders_gov_cronrunner_duplicate_fire_total[5m]) > 0warn#runbook-cronrunner-duplicate
CronRunnerBusFailuresrate(polytraders_gov_cronrunner_bus_failures_total[5m]) > 0page#runbook-cronrunner-bus-failure
CronRunnerDispatchLatencyHighhistogram_quantile(0.99, polytraders_gov_cronrunner_dispatch_latency_ms) > 500warn#runbook-cronrunner-latency

Dashboards

  • Grafana — Governance / CronRunner task dispatch timeline
  • Grafana — Governance / Scheduled task suppression rate

16. Developer Reporting

{
  "bot_id": "gov.cron_runner",
  "event_type": "CRON_TASK_DISPATCHED",
  "task_id": "hourly_health_sweep",
  "fired_at_ms": 1746788400000,
  "targets": [
    "gov.health_heartbeat"
  ],
  "lock_acquired": true,
  "trace_id": "tr_00deadbeef01"
}

17. Plain-English Reporting

SituationUser-facing explanation
Scheduled maintenance task firedThe system ran a scheduled internal task on time. No action needed.
Task skipped during quiet hoursA scheduled task was paused during a quiet period to avoid interference with market activity.
Task suppressed by kill switchScheduled trading-related tasks are paused. Governance and health checks continue.

18. Failure-Mode Block

main_failure_modeCronRunner process crashes or is partitioned from the internal bus, causing all scheduled tasks to miss their windows silently.
false_positive_riskA task fires twice if two CronRunner instances start simultaneously and both acquire the dispatch lock before expiry.
false_negative_riskA task is silently skipped if the system clock drifts more than the tick interval (1s), causing a cron boundary to be missed.
safe_fallbackOn restart, CronRunner reloads the manifest and resumes from the current tick. Missed ticks in the recovery window are not retroactively fired — governance bots that depend on scheduled triggers include their own staleness checks.
required_dependenciesConfig store (cron manifest), Internal message bus, Distributed lock (Redis or equivalent), KillSwitch status feed

19. Failure-Injection Recipes

ScenarioHow to injectExpected behaviourRecovery
REDIS_LOCK_UNAVAILABLETake Redis offline during a cron boundaryCronRunner falls back to in-process lock with WARN; CRON_RUNNER_DUPLICATE_FIRE suppression is weakened for the outage windowOnce Redis reconnects, distributed locking resumes. Any duplicate dispatches during outage are handled by downstream idempotency.
INVALID_MANIFEST_ENTRYPush a cron manifest entry with cron_expression='99 * * * *'CRON_RUNNER_INVALID_EXPRESSION raised; bad entry rejected; valid entries continue schedulingFix the manifest entry; reload triggers CRON_RUNNER_MANIFEST_RELOAD_OK.
BUS_PUBLISH_FAILUREDrop all packets to the internal bus for 60sDispatch attempts fail; CRON_RUNNER_BUS_PUBLISH_FAILED alert fired; tasks missed for that windowBus reconnects; next tick dispatches normally. Downstream bots handle missed ticks via staleness checks.
KILL_SWITCH_ACTIVE_DURING_TRADING_TASKActivate KillSwitch; observe behaviour at next hourly tick for trading-class tasksTrading tasks suppressed; governance and health tasks fire normally; KILL_SWITCH_ACTIVE loggedDeactivate KillSwitch; trading tasks resume at next tick.
DUPLICATE_CRONRUNNER_INSTANCESStart two CronRunner processes simultaneouslyOne instance acquires lock and dispatches; second emits CRON_RUNNER_DUPLICATE_FIRE and skipsShut down duplicate instance; primary continues normally.

20. State & Persistence

Cold-start recovery

On restart, cron manifest is re-fetched from config store and tickers are re-registered. Locks expire naturally. No missed ticks are retroactively fired.

21. Concurrency & Idempotency

AspectSpecification
Execution modelsingle-threaded event loop
Max in-flight100
Idempotency keytask_id + scheduled_at_ms (distributed lock)
Per-call timeout (ms)2000
Backpressure strategyskip dispatch and emit CRON_RUNNER_BUS_PUBLISH_FAILED if bus is saturated
Locking / mutual exclusionRedis SET NX PX per (task_id, scheduled_at_ms)

22. Dependencies

Depends on (must run first)

BotWhyContract
internal.config_storeCron manifest is loaded from the config store on startup and on change events.
internal.killswitchKillSwitch status is checked before dispatching trading-class tasks.

Emits to (downstream consumers)

External services

ServiceEndpointSLA assumedOn failure
Redis (distributed lock)99.99% (internal SRE target)

23. Security Surfaces

Abuse vectors considered

  • Injecting a malformed cron expression to crash the scheduler
  • Racing the distributed lock to trigger double-dispatch of financial tasks
  • Modifying the cron manifest to fire tasks outside of approved windows

Mitigations

  • All cron expressions are validated at manifest load; invalid entries are rejected with HARD_REJECT
  • Distributed lock (Redis SET NX) enforces single-dispatch per (task_id, scheduled_at_ms)
  • Config store changes are audit-logged; manifest reload events are emitted as OperationsReport entries
  • CronRunner has no CLOB or on-chain access — it only publishes to the internal bus

24. Polymarket V2 Compatibility

AspectValue
CLOB versionv2
Collateral assetpUSD
EIP-712 Exchange domain version2
Aware of builderCode fieldno
Aware of negative-risk marketsno
Multi-chain readyno
SDK usedinternal-only
Settlement contractnone
NotesCronRunner is internal-only and has no direct CLOB or on-chain interface. It dispatches triggers to other bots that operate on V2 surfaces.

API surfaces declared

internal

Networks supported

polygon

25. Versioning & Migration

FieldValue
spec2.0.0
implementation2.1.0
schema2
released2026-04-28

Migration history

DateFromToReasonAction taken
2026-04-28v1v2CLOB V2 cutoverNo direct CLOB changes required — CronRunner has no CLOB surface. Updated OperationsReport schema to pUSD denomination in downstream task payloads. Removed legacy USDC.e references from task manifest documentation.

26. Acceptance Tests

Unit Tests

TestSetupExpected result
Valid cron expression parses without errorcron_expression='0 9 * * *'Task scheduled for 09:00 UTC daily
Invalid cron expression raises ConfigErrorcron_expression='99 * * * *'ConfigError CRON_RUNNER_INVALID_EXPRESSION
Task suppressed during quiet hoursdisable_during_quiet_hours=true, now=quiet windowCRON_RUNNER_QUIET_HOURS_SKIP logged; no dispatch
Duplicate lock prevents double-fireTwo instances attempt to dispatch same task_id at same scheduled_at_msSecond instance emits CRON_RUNNER_DUPLICATE_FIRE and skips
KillSwitch suppresses trading tasksKillSwitch active; task target=strat.some_strategyTask suppressed; governance and health tasks fire normally

Integration Tests

TestExpected result
Task dispatched end-to-end on cron boundaryOperationsReport emitted; target bot receives trigger on internal bus
Manifest reload after config changeNew cron expressions take effect on next tick without restart

Property Tests

PropertyRequired behaviour
No task fires more than once per scheduled interval even under concurrent instancesAlways true — distributed lock enforces single dispatch
Every dispatch produces an OperationsReport heartbeatAlways true

27. Operational Runbook

CronRunner incidents fall into three categories: no dispatch (process down or bus failure), duplicate fires (lock race or multiple instances), or manifest errors (invalid expressions). All three block downstream governance tasks.

On-call actions

AlertFirst stepDiagnosisMitigationEscalate to
CronRunnerNoDispatchIn15mCheck CronRunner process status and internal bus connectivity.Governance pod lead
CronRunnerDuplicateFireRateCheck for multiple CronRunner instances; verify Redis lock is reachable.SRE on-call
CronRunnerBusFailuresCheck internal message bus health; review bus failure logs for task_id.SRE on-call
CronRunnerDispatchLatencyHighCheck Redis latency and internal bus throughput; reduce manifest size if overloaded.Governance pod lead after 30 minutes

Manual overrides

  • polytraders gov cron force-dispatch --task-id <id> — Manually trigger a missed cron task after investigating root cause.

Healthcheck

Endpoint: /internal/health/cron-runner | Green: Last dispatch < 65 minutes ago; manifest loaded; Redis lock reachable. | Red: No dispatch in 15 minutes; Redis unreachable; manifest parse errors.

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

GateHow measuredThreshold
Unit tests pass for cron expression parsing and lock deduplicationCI test run100% pass
Redis lock integration test verifiedIntegration testPass

Promote to Limited live

GateHow measuredThreshold
Dispatch latency p99 < 100ms under 20 simultaneous task firingspolytraders_gov_cronrunner_dispatch_latency_ms histogram< 100ms
Zero duplicate fires observed in 24h shadow runpolytraders_gov_cronrunner_duplicate_fire_total0

Promote to General live

GateHow measuredThreshold
End-to-end: all 97 bots receive their scheduled triggers on time for 7 consecutive daysGovernance audit log + OperationsReport timeline100% on-time dispatch
KillSwitch suppression verifiedFailure injection testPass

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%

RequirementStatus
Purpose defined✓ done
Required inputs listed✓ done
Parameters defined✓ done
Defaults defined✓ done
Warning thresholds defined✓ done
Hard thresholds defined✓ done
Safe fallback defined✓ done
Structured output defined✓ done
Developer log defined✓ done
Plain-English explanation✓ done
Unit tests defined✓ done
Integration tests defined✓ done
Property tests defined✓ done
Failure-mode block complete✓ done
Reference implementation pseudocode✓ done
Wire examples (input + output)✓ done
Reason codes listed✓ done
Metrics & logs defined✓ done
State & persistence defined✓ done
Concurrency & idempotency defined✓ done
Dependencies declared✓ done
Security surfaces declared✓ done
Polymarket V2 compatibility declared✓ done
Version & migration history declared✓ done
Operational runbook defined✓ done
Promotion gates defined✓ done
Failure-injection recipes defined✓ done