kaldamus entity reference · NewsArticle · NewsBrief · NewsDispatch

Scan, assess, dispatch

A rolling scan of the open web turns into one wire-desk article a day. What survives between the three stages isn't prose — it's a persisted trace an eval can recompute without trusting the model that wrote it.

Three entities, one direction of flow

Entity	id	Stage & payload
NewsArticle	`sha256(url)[:16]`	scan + classify — url/domain/provider, `what_happened` digest, `matches[]` (FactorMatch)
NewsBrief	UTC date	assess + rank — lede, ranked `stories[]` (each with factor_moves + `impacts[]`), gaps[]
NewsDispatch	= brief id (UTC date)	synthesize — headline/dek, link-free `body_source` → rendered `body_md`, ordinal citations[]

Concept

Example

Three strictly-sequential stages, each owning exactly one file: scan+classify → assess+rank → synthesize. Each reads only what the last stage persisted — never the one before it.

article → brief → dispatch

The dependency is one-directional: nothing downstream ever writes back upstream, and nothing upstream reads a later stage's output.

dispatch reads the brief, never re-assesses

A classified article asserts a factor-scale direction, never a probability — a closed-world FactorMatch against the existing factor vocabulary.

ten-year-treasury-yield · move −1.0 · conf 0.9

Scan → assess → dispatch

flowchart TD
    classDef srcNode fill:#16233b,stroke:#5a8fd4,stroke-width:1.5px,color:#dfe7f5;
    classDef briefNode fill:#241d0c,stroke:#c9a84c,stroke-width:1.5px,color:#f3e6bd;
    classDef detNode fill:#12151f,stroke:#5a6178,stroke-width:1px,color:#c9cbd6;
    classDef llmNode fill:#161822,stroke:#e0e0e0,stroke-width:2px,color:#f0f0f0;

    F["news_articles.json
+ factors.json, events.json"]:::srcNode --> S["Stage S — scan
keyless RSS + GDELT · no LLM"]:::detNode
    S --> C["Stage C — classify + match
LLM, batched · closed-world"]:::llmNode
    C --> ART["news_articles.json
matched + off-beat"]:::srcNode
    ART --> M["Stage M — impact graph
deterministic: factor→event edges"]:::detNode
    M --> A1["Stage A1 — assess
LLM qualitative ∥ deterministic pressure_logit"]:::llmNode
    A1 --> A2["Stage A2 — corroborate
lookup vs cached FactorObservation"]:::detNode
    A2 --> B["Stage B — compose
one LLM call: lede + link-free prose"]:::llmNode
    B --> BRIEF["news_briefs.json"]:::briefNode
    BRIEF --> D["Stage D — synthesize
separate job, reads the brief only"]:::llmNode
    ART -.cited articles.-> D
    D --> DISP["news_dispatches.json
replaces same-date row"]:::briefNode

Blue = the article corpus; gold = the two dated artifacts. Bright-bordered nodes are the three places an LLM actually writes: C classifies, B composes the brief, D synthesizes the dispatch — everything else is a lookup or arithmetic.

Concept

Example

Six lettered stages across two independently-runnable jobs — but only three call an LLM to write prose; the rest is deterministic arithmetic or a cache lookup.

6 stages, 3 LLM writes

Scan is deterministic and keyless — no LLM call, no per-article cost.

RSS + GDELT feeds → no cost

Classify is closed-world: Stage C can only emit a factor_id that already exists — an off-vocabulary article is still retained, just with an empty matches list, feeding the brief's gaps.

off-beat article → retained, no match

Corroboration never fetches — Stage A2 checks a story's moves against FactorObservation rows already on disk.

reuses the measure cache, no new call

Stage D is a second, independent job — it aborts loudly if no brief exists, and never sees a raw article beyond the deduped source list.

no brief for the date → hard abort

How state is managed

Row	Written by	Changes again when…
NewsArticle	the scan that first surfaces the URL	upserted in place if the same URL resurfaces (id = url hash); else ages out by retention
NewsBrief (whole)	one briefs run for that UTC date	never merged — a same-date rerun replaces the row outright
NewsDispatch (whole)	one dispatch run for that UTC date	upsert replaces by id — "double-fire replaces, never duplicates"

Concept

Example

BriefStory.prose is link-free by contract: it cites sources as ordinals, and the renderer joins ordinals → real URLs — so a fabricated citation is structurally impossible in the rendered artifact.

prose cites [1], never a URL

Parse-time validation rejects any http:// / https:// substring in the model's output outright — neither model ever has a channel to write a link.

a URL in body_source → rejected

Every write is whole-row replacement — a stricter posture than Factor's first-write-wins; no field survives a regeneration independent of the rest of the row.

regenerate → whole brief rebuilt

Transient, or permanent?

stateDiagram-v2
    direction LR
    classDef briefState fill:#241d0c,stroke:#c9a84c,stroke-width:1.5px,color:#f3e6bd;
    classDef deadState fill:#12151a,stroke:#3a4058,stroke-width:1px,color:#8892a4;

    [*] --> Scanned: first surfaced by a scan query
    Scanned --> Retained: accepted (matched or off-beat)
    Retained --> Retained: same URL re-surfaces (upsert)
    Retained --> Aged_out: fetched_at outside
article_retention_days at prune time
    Aged_out --> [*]

    class Retained briefState
    class Aged_out deadState

A NewsArticle row's lifecycle. Unlike Factor's "Orphaned" state, aging out is fully automatic — pruning runs at the end of every scan, no manual step.

Entity	Retention rule
NewsArticle	time window on `fetched_at`, pruned every run
NewsBrief	count cap — newest N by id kept
NewsDispatch	count cap — newest N by id kept

Concept

Example

None of the three behaves like events.json, which only ever grows — each has its own bounded retention rule.

age-window · count-cap · count-cap

Pruning is automatic, at the end of each run — there's no manual purge step the way orphaned factors need.

prune runs every scan

The audit chain: numbers that check themselves

Concept

Example

A quantitative impact isn't a model's assertion — it's four persisted numbers multiplied together, recomputable from the row alone without re-running any LLM.

0.5 × 0.35 × −1 × −1.0 = 0.175

The keystone check recomputes each contribution as salience × elasticity × sign × factor_move, sums them, and requires the total to match the stamped pressure_logit within a small epsilon.

impact_arithmetic_reproducible

Corroboration closes the loop to a source outside the pipeline: a story's moves are checked against cached FactorObservation rows — it either finds a fresh row or stamps no_data, never inventing an id.

no matching observation → no_data

The same closed-world discipline bounds what a story can claim: a FactorMatch can only name a Factor that already survived discovery — no pipeline mints a new causal node mid-story.

off-corpus factor_id → dropped by the parser

Scan, assess, dispatch

Three entities, one direction of flow

Scan → assess → dispatch

How state is managed

Transient, or permanent?

The audit chain: numbers that check themselves

See also