Five themes distilled from one month building, breaking, and re-architecting a local-LLM pipeline that turns prediction-market data into a navigable atlas. Less about what shipped — more about what the LLM taught us.
Local for volume, cloud for the narrow high-value seam
Size-matched Qwen3 runs summaries, scripts & prophecies on-device; Claude is retained for exactly one thing — an independent eval judge.
~$0 generation · $1–3 judge ceilingOne MLX model per process
Unified memory can't co-host two 4-bit 8B models. Each job loads one model and exits; regenerating a missing upstream artifact inline is forbidden.
Same-family judging is a correlated blind spot
Generate with Qwen, judge with Claude. The cross-family disagreement is itself the signal.
A cloud model judging another cloud model from the same family is a single point of correlated failure. Cross-family review is the whole point.
The interesting decision isn't local-vs-cloud as a stance — it's which work goes to which model. Generation is high-volume and cost-sensitive, so it runs on local Qwen3. Judging needs differential capability and independence, so it stays on Claude.
One-model-per-process is a physical constraint that shapes the whole pipeline: stages run in a fixed order, and a missing dependency is skipped loudly rather than recovered on the fly.
Local model quality is a moving target — the split is per-task, not a global "we're local now."
Logit decoding beats generate-then-parse for classification
Score the 10 canonical label tokens in one forward pass → softmax is the confidence. Can't emit off-list labels; confidence is real, not bucketed.
3 → 46 distinct confidences · +1pp accuracyPrompts are data, not code
Every prompt + sampler param + model id lives in a named, content-hashed GenerationConfig. A prompt edit is a leaf-level diff in the next eval.
Anchor format with one worked example, not five rules
For format, show don't tell. For distributional properties (date spread), show variance — not boundaries.
Defensive parsing + explicit forbidden patterns
Two-layer defense: the prompt forbids the chatter (soft), the parser strips it anyway (hard). Never rely on one alone.
Don't ask an LLM to choose between ten labels and then trust its self-reported confidence. Score the ten labels in one forward pass — that is the confidence.
A "freeform" LLM output is engineering debt with a friendly face. Each learning here narrows the space the model is allowed to produce — sometimes via decoding, sometimes via prompt, sometimes via a parser that tolerates the inevitable drift.
First-token scoring assumes labels diverge at token 1 — guard it with an init-time distinctness assertion that fails loudly.
Priority scoring replaces scattered exclusion gates
One priority_score: 0..100 per event. Each stage declares its own (min_score, max_items) — the cap bounds cost, the threshold biases which events get the budget.
One TTL knob, two layers of cache
A single ENTITY_CACHE_TTL_DAYS across all stages, plus an independent reuse check in the event generator. Either layer alone skips the work.
Mandatory-soft eval with a hard-fail escape hatch
The judge runs by default and soft-fails per row; --judge required hard-fails for the pre-merge gate.
A growing input shouldn't grow your bill. Cap absolute work; let the priority signal pick what makes the cut.
Local compute is "free" the way air is free — until you scale, at which point both run out. Cloud spend is loud and metered. Both want the same answer: do less work, but on the right items.
Separating "which items qualify" (threshold) from "how much budget exists" (cap) is what decouples cost from input growth — and it's what makes "run it every night" sustainable.
Soft nudges, not hard exclusions: a decisive, imminent event passes every gate — "if the market is screaming, there's probably a real story."
Stratified pass rates surface what averages hide
Every row carries probability band / market type / country bucket. A regression on scaled markets was invisible in the headline, obvious in the stratum.
Heuristics catch shape; the LLM judge catches semantics
Cheap deterministic checks own shape and fast-fail signals; the cross-family judge owns "is this a real historical event?" Don't ask the judge what a regex can answer.
Coverage floors complement orphan checks
For every "shouldn't exist" check, design the matching "should exist" check. Together they close the universe.
ConfigSnapshot — every run is a hash you can diff
Hash the entire decision surface, including the source of pure functions. "Did anything change?" becomes one --diff-config away.
A headline pass rate hides every regression that lives in a stratum. The stratum is where the regression actually is.
Every interesting failure in this project was either caught by eval or would have been, had eval been there earlier. The discipline is two-tiered and reproducible: slice every number by orthogonal axes, and pin the whole config to a content hash.
Per-config data worlds: --gen-config tighter materializes a complete parallel data/tighter/ — no flag-toggles, no shared state between A and B.
UNKNOWN deserves its own value, not a sentinel default
Optional[int]: None means "the market hasn't priced anything yet." Coercing no-signal to 50 was silently corrupting a fifth of the data.
Persist less than the LLM gives you
The factor prompt returns seven things; v1 stores one. The rest is reasoning fuel — legitimate to ask for, fine to discard.
First-write-wins on shared LLM-generated entities
Freeze the descriptive payload at first discovery; only counters and timestamps update. Drift goes from inevitable to opt-in.
Upstream change → explicit downstream drop
TTLs handle time, not meaning. A category change drops the stale summary & script explicitly, forcing regeneration.
Coercing "no signal" to a default value is silent data corruption. Make the unknown legible — in the type, in the prompt, in the UI.
The interesting tradeoffs live in the gap between what the model returns and what the product persists: how to encode "I don't know," how much to keep, and when to freeze what the model said.
Stages stay independently restartable, one JSON file at a time — the filesystem is the interface, every failure partial-and-resumable.
Local vs cloud is per-task, not global. Generation can be local and free; judging needs to stay cross-family and metered. The same project can be both.
Two-layer defense beats clever single-layer. Prompt forbids the pattern + parser strips it. Heuristic checks shape + judge checks semantics. Worker reuses cache + consumer reuses cache. The redundancy is the design.
Identity by content hash, not by name. Configs collapse on hash, snapshots dedupe on hash, asset SHA256s pin data state. Every "did anything change?" becomes one comparison.
Cap absolute work; let priority pick. max_items bounds cost regardless of input growth; min_score biases what makes the cut. Independent knobs — you usually want both.
Make the unknown legible. In the type, the prompt, the UI, the eval. Silent defaults are the worst failure mode an LLM pipeline can have.
Less about what shipped — more about what the LLM taught us.
Five weeks, 17 major sessions, one local-LLM pipeline turning prediction-market data into a navigable atlas. The throughline: the model is a component with a failure surface, and engineering discipline is what makes that surface legible.
Source: kaldamus / CHANGELOG.md · PROJECT.md · Apr 29 – May 30, 2026 · 17 sessions distilled.