kaldamus entity reference · Prediction → Event

Ingestion: Prediction → Event

Every event begins as one raw Kalshi market row. This is the transform that turns it into the enriched, permanent record the rest of the pipeline reads — and the one timestamp the whole freshness model is anchored on.

One row becomes one row

Prediction · raw	how it transforms	Event · enriched
`id` · `series_ticker`	carried verbatim	`id` · `series_ticker`
`title` + `sub_title`	joined as `"{title} — {sub_title}"`	`title`
`mutually_exclusive` + market count	≤1 → binary · flag → exclusive · else scaled	`market_type`
`markets[]` one dict per outcome	one Outcome each; leading market's price → probability	`outcomes[]` · `probability`
`markets[0]`	rules & resolution profile, capped	`rules` · `rules_secondary` · `opened_at`
`markets[*]` prices + volume	summed; bid/ask read off the leading market	`signals`
`category` raw Kalshi	gate only — then discarded	— classifier sets `enrichment.category`
`loaded_at`	fetch timestamp	— not persisted

Inside each raw market: prices (yes_bid/yes_ask, previous_price) · volume (volume, open_interest, liquidity) · rules (rules_primary, rules_secondary).

Concept

Example

One Prediction becomes exactly one Event — a transform, not a join. Only id and series_ticker survive unchanged; everything else is recomputed from markets[] on every run.

KXSPEAKER-26 → Event(id="KXSPEAKER-26")

A prediction with more than five nested markets never becomes an Event — the complexity gate drops it first.

a 7-market bracket → no Event

The raw category only gates sports out; the Event's category is the classifier's, and the two often disagree.

raw "Politics" → enriched "Legal"

How the two jobs run

flowchart TD
    classDef pred fill:#16233b,stroke:#5a8fd4,stroke-width:1.5px,color:#dfe7f5;
    classDef work fill:#241d0c,stroke:#c9a84c,stroke-width:1.5px,color:#f3e6bd;
    classDef dec fill:#12151f,stroke:#5a6178,stroke-width:1px,color:#c9cbd6;
    classDef side fill:#12151a,stroke:#3a4058,stroke-width:1px,color:#8892a4,stroke-dasharray:3 3;
    K["Kalshi /events?status=open"] --> G{"raw category == Sports?"}:::dec
    G -- yes --> Z["dropped — no Prediction row"]:::side
    G -- no --> PJ["predictions.json
save_all overwrite"]:::pred
    PJ --> S1{"enrichment cache fresh?"}:::dec
    S1 -- fresh --> EJ["reuse cached category + geo"]:::work
    S1 -- "stale / forced" --> S1C["classify + geo worker"]:::work
    S1C --> EJ
    EJ --> S2["process_events()
market_type · probability · priority"]:::pred
    S2 --> UP["EventsRepo.upsert()
never deletes"]:::pred
    UP --> REC["reconcile()
departed→closed, resolved→settled"]:::pred
    REC --> EVJ["events.json"]:::pred

One run of update_predictions then update_events. Blue = the data path; gold = the enrichment worker; dashed = side effects.

Concept

Example

Two jobs, three stages — fetch and gate, enrich, then transform. Only the fetch stage touches the network for market data.

nightly: update_predictions → update_events

A plain run replaces the whole predictions file; only a scoped run keeps the existing rows.

full run vs --id KXSPEAKER-26

The resolution lookup is separate and best-effort — capped per run, and it fails quietly.

network error → predictions still saved

Stage 2 trusts the enrichment worker's output as-is; the worker owns its own freshness.

a 7-day-old category is reused unchanged

A changed category invalidates that event's summary and script in the same run.

"Economics" → "Politics" ⇒ summary + script dropped

Reconciliation runs only on full passes, so a scoped run can't mass-close events.

an --id run → other statuses untouched

The cascade-root invariant

stateDiagram-v2
    classDef openState fill:#16233b,stroke:#5a8fd4,stroke-width:1.5px,color:#dfe7f5;
    classDef closedState fill:#231a0c,stroke:#c98500,stroke-width:1.5px,color:#f0c968;
    classDef settledState fill:#12151a,stroke:#3a4058,stroke-width:1px,color:#8892a4;
    [*] --> open: stamped across the fresh open set
    open --> closed: departed, no resolution yet
    open --> settled: departed, resolution found
    closed --> settled: resolution arrives on a later run
    settled --> [*]
    closed --> [*]
    class open openState
    class closed closedState
    class settled settledState

Lifecycle is reconciled on full runs only. There is no path back to open, and no path out of the file once a row is terminal.

Concept

Example

event.enriched_at is the enrichment worker's own timestamp — not the moment the job last touched the row. Every downstream freshness check is anchored to it.

enriched_at: 2026-07-25, kept on an 08-01 re-run

A fresh wall-clock time is written only when an event has no enrichment record at all.

brand-new event → enriched_at = now

Stamp it on every touch and stale enrichments read as fresh — the cascade stops catching them.

summary 07-30 vs enriched 07-25 ⇒ fresh

A multi-market settlement has no single overall YES/NO, so its outcome stays empty by design.

a 5-way race settles → resolved_outcome: null

Transient, or permanent? An inversion

Concept

Example

The raw snapshot that looks like the source of truth is disposable; the derived, enriched record is what persists.

predictions.json rewritten · events.json appended

The raw mirror is wholesale-replaced each run — being the direct Kalshi copy earns it no permanence.

today's fetch overwrites yesterday's file

The enriched record is only ever added to; no code path removes a row from it.

a settled event from months back is still present

A file that only ever grows

Concept

Example

events.json has no retention policy — nothing deletes a settled or closed row, and the assembled event graph grows in lockstep.

closed & settled rows never leave the file

The event graph unions every node on each rebuild, so it tracks that growth one-for-one.

graph.json nodes = events.json rows

Ingestion: Prediction → Event

One row becomes one row

How the two jobs run

The cascade-root invariant

Transient, or permanent? An inversion

A file that only ever grows

See also