kaldamus entity reference · EvalReport · QualityReport · ConfigSnapshot

EvalReport, QualityReport, ConfigSnapshot

One nightly run scores the pipeline against itself and writes three different kinds of row — one keeps a bounded history, one keeps none of its own, and one can never legally change.

Three entities, three jobs

Entity	id	Permanence model
EvalReport	`entity_type_run_id`	bounded append-only history — the raw per-entity scoring output
QualityReport	`"current"`	singleton, no history of its own — a distilled, human-friendly view
ConfigSnapshot	`sha256(payload)[:16]`	immutable, content-addressed — the config-trio receipt

Concept

Example

One eval run touches all three: EvalReport is the raw per-entity scoring, QualityReport a distillation of it, ConfigSnapshot a receipt for the config the scoring ran against.

1 run → up to 14 EvalReport rows + 1 snapshot

QualityReport never scores anything itself — it re-reads whatever EvalReport history is on disk and re-renders it.

distills, doesn't compute

ConfigSnapshot belongs to no single entity — captured once per run, then referenced by every EvalReport row that run writes.

one snapshot shared across the run's rows

How update-reports runs

flowchart TD
    classDef srcNode fill:#16233b,stroke:#5a8fd4,stroke-width:1.5px,color:#dfe7f5;
    classDef reportNode fill:#241d0c,stroke:#c9a84c,stroke-width:1.5px,color:#f3e6bd;
    classDef decisionNode fill:#12151f,stroke:#5a6178,stroke-width:1px,color:#c9cbd6;
    classDef llmNode fill:#161822,stroke:#e0e0e0,stroke-width:2px,color:#f0f0f0;

    A["make update-reports"]:::srcNode --> B["capture ConfigSnapshot
id = sha256(canonical_json)[:16]"]:::reportNode
    B --> C["config_snapshots.json
never pruned"]:::reportNode
    B --> D{"for each ENTITY_TYPE"}:::decisionNode
    D --> E["sample rows
stratified"]:::srcNode
    E --> F["deterministic checks + gates
$0"]:::srcNode
    F --> G{"judge covers
this entity?"}:::decisionNode
    G -- "yes (narratives, factors,
forecasts, deps, news…)" --> H["Claude rubric judge
soft-fail per row"]:::llmNode
    G -- "no (events, history,
classifier, geo, gates)" --> I
    H --> I["EvalReport row
id = entity_type_run_id"]:::reportNode
    I --> J["eval_reports.json
append, then prune_to_runs()"]:::reportNode
    J -.->|"next entity_type"| D
    J --> K["_build_quality() — re-reads ALL
retained rows per entity_type"]:::srcNode
    K --> L["QualityReport singleton
id='current', full overwrite"]:::reportNode

Gold = a persisted row (report or snapshot); blue = in-flight work. The judge branch is the only step that costs money; everything else in the loop is $0.

Concept

Example

One config capture, then one pass per entity type — sample, check, optionally judge, write — then a second job reads the whole history back and distills it.

capture → score ×N → distill

Sampling is per entity type, not global — narrative entities stratify a sample; classifier/geo re-run a fixed fixture; gates assert whole-corpus invariants.

gates samples nothing

The judge branch is the only step that costs money — checks and gates are $0, and a judge soft-fails per row rather than aborting the run.

judge unavailable → error: judge_failed, run continues

The ConfigSnapshot is captured once, before any entity is scored, and its id is stamped onto every EvalReport row that run.

all the run's rows share one config_snapshot_id

Every write mints a brand-new row; nothing overwrites an existing EvalReport. A prune sweep drops the oldest run's rows once the window is exceeded.

prune_to_runs() after the batch

How state is managed

Row	Written by	Changes again when…
EvalReport row	one run, per entity_type	never — a new run mints a brand-new row; the old one sits until pruned
eval_reports.json window	`prune_to_runs()`, after every write batch	drops the oldest run_id's rows once more than `retain_runs` distinct runs are on file
QualityReport (id="current")	every run, via `_build_quality()`	overwritten in full, same id, every run — never incremented
ConfigSnapshot row	the first run scored against a given trio	never — the id is a hash of the row's own content

Concept

Example

Four write regimes for four rows — none of them is "upsert in place": mint, prune, overwrite, content-address.

a new run always mints, never edits

Content-addressing is the strongest immutability in the set: a snapshot's id is a hash of its body, so you can't mutate a row — only create a different one.

edit one field → different id

Two consecutive runs with no config change share the same snapshot id and write nothing new at all.

unchanged trio → no new row

Nothing prunes config_snapshots.json, so it accumulates independently — a snapshot can outlive every EvalReport that ever cited it.

old snapshot → kept, unreferenced

Transient, or permanent?

	EvalReport	QualityReport	ConfigSnapshot
Deleted by	`prune_to_runs()` — automatic	nothing — overwritten, not deleted	nothing routine — no prune exists
History kept	bounded append-only: a retain window of runs × entity types	none of its own — recomputed live from EvalReport's window	full history since first capture — unbounded
Mutability	append-only while retained, then hard-deleted in a batch	mutable in place — same id, new content	can't change without becoming a different id

Concept

Example

All three permanence answers in one family — a bounded ledger, a view, and an immutable receipt. Nothing else in the doc set packs three models into one stage.

remember-a-while · recompute · never-change

EvalReport remembers, but only as long as retain_runs says; QualityReport is a view recomputed wholesale every run, with no memory of its own.

QualityReport = live re-render of the ledger

ConfigSnapshot is the only one where "permanent" isn't a policy someone could quietly change — it's a property of what a sha256 hash is.

immutability by construction, not convention

The largest file

Concept

Example

eval_reports.json is the largest file any scored, generated entity writes — because it's the only entity family in the pipeline that's genuinely append-heavy.

a new row every night, per entity type

Every other generated entity is either a singleton (overwritten, never grows) or capped well below eval's row count.

singletons vs a growing ledger

The ceiling is bounded, not open-ended: entity types × retained runs — append-heavy, but not unbounded the way events.json is.

ENTITY_TYPES × retain_runs

Row size is uneven — the factors row alone is a large share, because it samples at the full cap, carries a multi-dimension judge verdict, and adds corpus-level blocks no other entity does.

factors row ≫ a news_dispatches row

A missing run reads differently by position: only the oldest missing is a suite-growth artifact; a run missing from the middle is a one-off failure worth watching.

middle gap → check for recurrence

EvalReport, QualityReport, ConfigSnapshot

Three entities, three jobs

How update-reports runs

How state is managed

Transient, or permanent?

The largest file

See also