4.1 NewsIngest

1. Bot Identity

2. Purpose

NewsIngest continuously pulls news from external feeds (RSS, partner APIs), resolves named entities to Polymarket market IDs via the Gamma API, and scores each story for materiality. High-materiality stories trigger immediate ObservationReport emission; routine items are sampled 1/10. NewsIngest is strictly read-only — it never submits, signs, or modifies orders. Output is consumed by news-materiality-trader and contradictiondetector.

3. Why This Bot Matters

• High-materiality story missed

  news-materiality-trader fails to re-price after a major event; positions held through adverse resolution.

• Entity resolution maps story to wrong market

  Materiality signal sent to unrelated market, polluting downstream strategies with false signal.

• Duplicate story flooding the bus

  Contradictiondetector and materiality-trader overwhelmed with redundant events, latency spikes.

• Stale feed accepted as fresh

  Old news recycled as new; obsolete materiality score triggers trades on stale information.

No worked examples on this bot yet. Worked examples are optional but strongly recommended — they turn an abstract failure mode into something a developer can verify in a fixture.

4. Required Polymarket Inputs

5. Required Internal Inputs

6. Parameter Guide

7. Detailed Parameter Instructions

{ "materiality_min": 0.3 }

if (story.score < params.hard) drop('NEWSINGEST_BELOW_MATERIALITY_FLOOR');

{ "dedup_window_s": 300 }

if (p.dedup_window_s < p.hard) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL');

{ "source_priority": ["reuters", "ap", "bloomberg", "league_feeds", "official_handles"] }

// not yet authored

{ "emit_every_above": 0.7 }

if (p.emit_every_above < p.hard) throw ConfigError('PARAMETER_CHANGE_REQUIRES_APPROVAL');

8. Default Configuration

{
  "bot_id": "intel.newsingest",
  "version": "2.1.0",
  "mode": "general_live",
  "defaults": {
    "materiality_min": 0.3,
    "dedup_window_s": 300,
    "source_priority": [
      "reuters",
      "ap",
      "bloomberg",
      "league_feeds",
      "official_handles"
    ],
    "emit_every_above": 0.7
  },
  "locked": {
    "dedup_window_s": {
      "min": 10
    },
    "emit_every_above": {
      "min": 0.4
    }
  }
}

9. Implementation Flow

1. Subscribe to all configured external feeds (RSS polling + partner API long-poll or webhook).
2. On each incoming story: compute a 64-bit fingerprint (source + headline hash). Check dedup_window_s ring buffer; if duplicate, drop silently.
3. Perform entity extraction (NER); resolve entities against Gamma API market metadata to produce a list of candidate condition_ids.
4. If no condition_id matched, drop story with NEWSINGEST_NO_MARKET_MATCH.
5. Score materiality (0–1) based on: source priority weight × entity confidence × recency decay × neg-risk amplifier if any matched market has neg_risk=true.
6. If score < materiality_min hard floor, drop with NEWSINGEST_BELOW_MATERIALITY_FLOOR.
7. Check KillSwitch; if active, ingest and score but suppress ObservationReport emissions.
8. Apply sampling rule: if score ≥ emit_every_above, emit every time; else sample 1/10.
9. Emit ObservationReport with: report_id, trace_id, condition_ids, materiality_score, story_summary, source, pub_ts_ms, neg_risk_flag, and sampling_applied.
10. Log ingest cycle summary: stories_received, deduped, no_match, below_floor, sampled_out, emitted.

10. Reference Implementation

Subscribes to external feeds, fingerprints each story for dedup, resolves entities to Polymarket condition_ids via Gamma API, scores materiality, applies sampling, and emits ObservationReport for stories that pass all gates.

Pseudocode is language-agnostic. FETCH = read input. EMIT = produce output.

FUNCTION ingestFeed(feed_event):
  // --- 0. Dedup ---
  fp = fingerprint(feed_event.source + feed_event.headline)
  IF fp IN dedup_ring_buffer(dedup_window_s):
    LOG DEBUG 'NEWSINGEST_DEDUP_DROP'
    RETURN
  dedup_ring_buffer.add(fp, ttl=dedup_window_s)

  // --- 1. KillSwitch gate ---
  ks = FETCH internal.killswitch.status
  killswitchActive = ks.active

  // --- 2. Entity extraction + market resolution ---
  entities = extractEntities(feed_event.body)
  condition_ids = []
  FOR entity IN entities:
    markets = gamma_api.GET('/markets?keyword=' + entity.canonical)
    FOR m IN markets:
      IF m.active AND NOT m.closed:
        condition_ids.append(m.condition_id)

  IF condition_ids IS EMPTY:
    LOG DEBUG 'NEWSINGEST_NO_MARKET_MATCH'
    RETURN

  // --- 3. Materiality scoring ---
  source_weight = SOURCE_WEIGHTS[feed_event.source] OR 0.5
  entity_conf   = AVG(entity.confidence FOR entity IN entities)
  recency_decay = exp(-age_s / 600)         // half-life 10 min
  neg_risk_amp  = 1.2 IF any(m.neg_risk FOR m IN markets) ELSE 1.0
  score = source_weight * entity_conf * recency_decay * neg_risk_amp

  IF score < params.materiality_min.hard:
    LOG DEBUG 'NEWSINGEST_BELOW_MATERIALITY_FLOOR'
    RETURN

  // --- 4. Warning annotation ---
  warnings = []
  IF score < params.materiality_min.default:
    warnings.append('NEWSINGEST_LOW_MATERIALITY')

  // --- 5. Sampling ---
  IF score >= params.emit_every_above:
    sampling_applied = false
  ELSE:
    IF random() > 0.1:                      // sample-1/10
      RETURN
    sampling_applied = true

  // --- 6. KillSwitch suppress ---
  IF killswitchActive:
    LOG INFO 'KILL_SWITCH_ACTIVE — suppressing ObservationReport'
    RETURN

  // --- 7. Emit ---
  report = ObservationReport(
    report_id        = 'rep_ni_' + condition_ids[0][:6] + '_' + now_ms(),
    trace_id         = newTraceId(),
    bot_id           = 'intel.newsingest',
    kind             = 'ObservationReport',
    condition_ids    = condition_ids,
    materiality_score= score,
    story_summary    = feed_event.headline[:200],
    source           = feed_event.source,
    pub_ts_ms        = feed_event.pub_ts_ms,
    neg_risk_flag    = neg_risk_amp > 1.0,
    sampling_applied = sampling_applied,
    warnings         = warnings,
    emitted_at_ms    = now_ms()
  )
  EMIT internal.bus.observations <- report

SDK calls used

• gamma_api.GET('/markets?keyword=<entity>')
• gamma_api.GET('/market/<condition_id>')

Complexity: O(E × M) per story, where E = extracted entities, M = live markets per entity

11. Wire Examples

Input — what arrives on the wire

{
  "label": "Incoming AP wire story",
  "source": "ap",
  "payload": {
    "source": "ap",
    "headline": "Federal Reserve holds rates at 4.25–4.50% at May 2026 meeting",
    "body": "The Federal Open Market Committee voted unanimously to hold...",
    "pub_ts_ms": 1746700800000,
    "feed": "rss"
  }
}

Output — what the bot emits

{
  "label": "ObservationReport — high-materiality story",
  "payload": {
    "report_id": "rep_ni_0xabc1_1746700800000",
    "trace_id": "trc_0xdeadbeef01020304050607",
    "bot_id": "intel.newsingest",
    "kind": "ObservationReport",
    "condition_ids": [
      "0xabc1230000000000000000000000000000000000000000000000000000000000"
    ],
    "materiality_score": 0.82,
    "story_summary": "Federal Reserve holds rates at 4.25–4.50% at May 2026 meeting",
    "source": "ap",
    "pub_ts_ms": 1746700800000,
    "neg_risk_flag": false,
    "sampling_applied": false,
    "warnings": [],
    "emitted_at_ms": 1746700800452
  }
}

12. Decision Logic

13. Standard Decision Output

This bot returns a ObservationReport object. See ObservationReport schema.

{
  "report_id": "rep_ni_0xabc1_1746700800000",
  "trace_id": "trc_0xdeadbeef01020304",
  "bot_id": "intel.newsingest",
  "kind": "ObservationReport",
  "condition_ids": [
    "0xabc1230000000000000000000000000000000000000000000000000000000000"
  ],
  "materiality_score": 0.82,
  "story_summary": "AP: Fed holds rates unchanged at May 2026 meeting",
  "source": "ap",
  "pub_ts_ms": 1746700800000,
  "neg_risk_flag": false,
  "sampling_applied": false,
  "warnings": [],
  "emitted_at_ms": 1746700800450
}

14. Reason Codes

15. Metrics & Logs

Metrics emitted

Alerts

Dashboards

• Grafana — Intelligence / NewsIngest feed health
• Grafana — Intelligence / materiality score distribution

16. Developer Reporting

{
  "bot_id": "intel.newsingest",
  "cycle_ts_ms": 1746700800000,
  "stories_received": 47,
  "deduped": 12,
  "no_market_match": 8,
  "below_floor": 5,
  "sampled_out": 19,
  "emitted": 3,
  "top_materiality": 0.82,
  "killswitch_active": false
}

17. Plain-English Reporting

18. Failure-Mode Block

19. Failure-Injection Recipes

20. State & Persistence

Cold-start recovery

On cold start, dedup buffer is empty; the first dedup_window_s of operation may emit some near-duplicates from wire re-transmissions.

21. Concurrency & Idempotency

22. Dependencies

Depends on (must run first)

Emits to (downstream consumers)

External services

23. Security Surfaces

Abuse vectors considered

• Malicious external feed injecting a crafted headline that resolves to a high-value condition_id with artificially elevated materiality score
• Entity resolution poisoning — an entity alias added to the watchlist that maps news about an unrelated topic to a targeted market

Mitigations

• condition_id format validated against known 32-byte hex pattern before inclusion
• Gamma API responses cross-checked against local market ID whitelist
• All ObservationReports are recommendations only — downstream guardrails independently re-validate

24. Polymarket V2 Compatibility

API surfaces declared

Networks supported

25. Versioning & Migration

Migration history

26. Acceptance Tests

Unit Tests

Integration Tests

Property Tests

27. Operational Runbook

NewsIngest incidents are usually external feed outages or Gamma API slowness. The bot is read-only so incidents do not affect active positions — only new signal discovery.

On-call actions

Manual overrides

• —

Healthcheck

GET /internal/health/newsingest -> 200 if Last story received < 60 s ago AND Gamma API responding AND dedup buffer writable. RED if All feeds silent > 120 s OR Gamma API errors > 50% for 5 min.

28. Promotion Gates

A bot does not advance to the next readiness state until every gate below is green. Gates are observable from production data — no subjective sign-off.

Promote to Shadow

Promote to Limited live

Promote to General live

29. Developer Checklist

Ready-to-ship score: 27/27 sections complete · 100%