kaldamus entity reference · SummaryReport · BreaksReport

Pipeline diagnostics

Every make update-all run ends by asking two blunt questions of itself — what's in here, and what fell through the cracks — and answers both in pure Python, with no LLM call, no network, and no memory of the run before this one.

Two snapshots, one job, no join

Entity	Answers	Holds
SummaryReport	"what exists"	entity_counts · per-category / per-status breakdowns · factor_dedup_ratio · Timeliness block · stale_counts
BreaksReport	"what's missing, and why"	each gap split into `_expected` (a gate explains it) / `_unexpected` (nothing does)

Concept

Example

Two sibling diagnostic snapshots from one job, one run, one now — SummaryReport counts what exists, BreaksReport counts what's missing. Nothing on one row points at the other.

both id="current", no join

BreaksReport splits each gap by a gate predicate: _expected carries {id, category, reason}; _unexpected is a bare id list — a real drop no gate explains.

gate fires → _expected · else _unexpected

This exists separately from the eval suite: evals score the quality of what was produced; these surface counts and unaccounted-for gaps in a scannable form.

"what's missing and why", not pass/fail

How update-reports runs

flowchart TD
    classDef srcNode fill:#16233b,stroke:#5a8fd4,stroke-width:1.5px,color:#dfe7f5;
    classDef reportNode fill:#241d0c,stroke:#c9a84c,stroke-width:1.5px,color:#f3e6bd;
    classDef gateNode fill:#12151f,stroke:#5a6178,stroke-width:1px,color:#c9cbd6;
    classDef writeNode fill:#161822,stroke:#e0e0e0,stroke-width:2px,color:#f0f0f0;

    A["every upstream entity file
read via the canonical repos"]:::srcNode --> N["now = datetime.now(timezone.utc)
one timestamp for the whole run"]:::writeNode
    N --> S["_build_summary()
counts, breakdowns, Timeliness"]:::srcNode
    N --> B["_build_breaks()
one predicate per gap type"]:::srcNode
    B --> G{"gate fires?
priority_score < min_score"}:::gateNode
    G -- "yes — a policy explains it" --> EXP["_expected:
{id, category, reason}"]:::reportNode
    G -- "no — nothing explains it" --> UNEXP["_unexpected:
bare id"]:::reportNode
    S --> W["save_all(...)
id='current', overwrite, no history"]:::writeNode
    EXP --> W
    UNEXP --> W
    W --> O["summary_report.json
breaks_report.json"]:::reportNode

Every run of update_reports.py. Blue = raw entity files coming in; gold = the two report files going out; the gate diamond is the whole design point of BreaksReport.

Concept

Example

One pure-Python pass — no LLM, no network: read every upstream entity file via the canonical repos, compute at one timestamp, split each gap by a gate, overwrite both singleton rows.

reads on-disk files → writes 2 rows

One now threads through every age bucket and staleness check, so the whole report describes one consistent instant.

datetime.now() captured once

Gates are imported from config/categories.py, never re-derived — a new gate must be applied here in the same change, or gated drops surface as unexpected.

EXCLUDED_RAW_CATEGORIES imported live

--category / --id are accepted for CLI uniformity, then ignored — the reports are gen-config-scoped aggregates with no per-event variant.

--id passed → no effect

How state is managed

Concept

Example

There's no state machinery — by design. Every field is fully recomputed at one now, from files this job doesn't own, against gates it doesn't define.

one transition ever: replaced

Singleton, no history: both rows overwrite every run. If you ever need history, switch to an EvalReport-style (run_id, retain) shape — don't shoehorn a list of singletons.

id="current", overwritten

The reports never assert quality — no coverage floors, no pass/fail; that's the eval gates' job. Descriptive, not evaluative.

counts, not scores

The one thing they can drift against is the gate logic, if config/categories.py changes without a matching change here.

new gate, un-mirrored → false unexpected

Transient, or permanent?

	SummaryReport / BreaksReport	EvalReport
Shape	singleton — one row, `id="current"`	composite — one row per (entity_type, run_id)
History	none — every run overwrites the one row	append-only, bounded by a retain cap
Content	descriptive — counts, breakdowns, gaps	evaluative — check pass rates, judge dims, scores
Answers	"what's true right now?"	"is quality better or worse than last run?"

Concept

Example

Singleton-no-history by design — and the doc set's other diagnostic-shaped entity, EvalReport, makes the opposite choice on purpose.

overwrite vs append

A count or a gap is a fact about the current corpus — there's no meaningful run-over-run comparison, since the report already gives the current gap directly.

this run's gap stands alone

A quality score is a run-over-run question ("did this run regress?") — which you can't ask of a single overwritten row, so EvalReport is keyed on run_id and retained.

regression needs history

Reading the live breaks: excluded on purpose, or actually broken?

Concept

Example

An _expected count is the pipeline working as designed; an _unexpected count is the only number here worth losing sleep over.

0 unexpected → the healthy case

Even a nonzero unexpected count can resolve cleanly one layer deeper — because the breaks gate is a simplification of the real selector.

unexpected gap → often explained by the selector

The gate encodes only the score threshold — the rule config/categories.py-style gates can express — but the real selector also applies a spend cap and a terminal-status skip.

max_items cap + settled-skip not in the gate

So a score-qualifying event ranked outside the budget, or a settled one, reads as "unexpected" with no broken call behind it.

rank beyond max_items → dropped, not broken

Pipeline diagnostics

Two snapshots, one job, no join

How update-reports runs

How state is managed

Transient, or permanent?

Reading the live breaks: excluded on purpose, or actually broken?

See also