A five‑week engineering retrospective: cost cutovers, prompt evolution, classifier calibration, probability semantics, and the eval framework that surfaced every bug we missed.
Kaldamus transforms the Kalshi prediction‑market firehose into an interactive atlas of narratives. Seven batch jobs run a pipeline from raw markets through LLM‑generated summaries, anchor scripts, Nostradamus prophecies, and causal‑factor decomposition. All entities are immutable JSON files keyed by run config.
The frontend is vanilla JS with six tabs. It runs identically against a FastAPI dev server or a static GitHub Pages bundle — one source tree, two deployment modes.
The deliberate constraints (no DB, no auth, no framework, no multi‑tenancy) turned out to be the system's biggest accelerator: every change is local, every artifact is a file you can git diff.
Aggressive scope constraints — not architecture cleverness — were the lever that let a one‑person team ship and iterate at API velocity.
Kalshi exposes thousands of binary, exclusive, and scaled markets across politics, finance, science and culture. Each is a real‑money probability — but without titles parsed, geography extracted, or causal context, it is a wall of tickers.
Four overlapping audiences want different cuts of the same data: forecasters want narrative, researchers want causal indicators, leadership wants bounded cost, and the editorial use case wants ready‑to‑read scripts. One pipeline serves all four because they share a substrate — the event — with different downstream artifacts.
The product is not a dashboard. It is signal extraction plus narrative: probability is the input, prose and structured factors are the output.
A market price is data; a story about why is the product. Multiple audiences sharing one event substrate justified one pipeline with many heads.
The first working version routed every generation task through the Anthropic API. Summaries, anchor scripts, Nostradamus prophecies, and quality judging all hit the same external surface. Enrichment was a separate HTTP service.
The assumption was simple: Claude is reliable, the prompts are short, and the per‑event spend is acceptable while the system is small. The hidden assumption was that the system would stay small.
What looked like a clean architecture had two latent failure modes: cost grew with the universe of Kalshi events (no upper bound), and latency depended on a remote queue outside our control. Neither hurt at 50 events. Both hurt at 2,500.
Cloud‑first architectures implicitly assume the workload stays bounded. When the upstream universe grows, the per‑event cost model breaks before the architecture does.
Every assumption in the left column survived to production. Each was eventually contradicted by data that already existed in the pipeline — we just had not built the check that would have surfaced it.
The pattern: silent compensating behavior. A truthy‑zero check returns the fallback. A sum‑clamp swallows over‑summing exclusives. A garbage‑bin label catches off‑list classifier outputs. Each defensive line of code prevented a crash and obscured a bug.
The fix was structural, not heuristic: build observability for the silent paths, then let the data tell you which defensive default was hiding a real signal.
Defensive defaults that prevent crashes also prevent discovery. Treat every fallback as a future bug until a check explicitly measures the rate at which it fires.
Prompts moved through three concrete phases. They began as config/prompts.py literals tuned for Claude. The local‑LLM cutover forced explicit "no preamble, no markdown fences, no commentary" rules plus tolerant parsers (_parse_json_object, _strip_chatty_preamble) to absorb Qwen verbosity.
The decisive move was externalizing prompts + model ids + sampler params into workers/generation/configs/<name>.json. A new data_path(filename, config_name) resolver gave each config its own folder. Running --gen‑config tighter now produces a parallel world without any cp shuffles or cache‑TTL hacks.
The lesson from 2026‑05‑07 reinforced this: three prompt edits on a 1.5B classifier were net‑neutral or worse. Capacity, not wording, drove the leap from 0.75 to 0.94 accuracy on a 4B model.
Externalize prompts the day you have two of them. Prompt‑tuning small models is noise; A/B at the config level is signal.
The unit of context is one event: title, outcome titles, probability, close year. That is small enough — usually well under 1k tokens — that a 4‑bit 7–8B local model is competitive with cloud Claude on narrative tasks.
Each job loads its model once, then runs a stateless generate() per event. Output goes through a tolerant parser. A sidecar generation_stats.json records gen_ms and gen_tokens per artifact, surfacing mean + p95 in the EvalReport — the early-warning system for prompt bloat.
The invariant that only one MLX model lives in a process shaped the pipeline: scripts depend on summaries, but a script job will never on‑the‑fly generate a missing summary. It logs a warning and continues. Order in make update-all is contract, not coincidence.
Event‑scoped context made local 7B viable. The per‑process model invariant is what kept the system simple under load.
Kaldamus does not run a vector store. Two observations made retrieval unnecessary for the narrative tasks: per‑event context is small, and the prose is judged on style and faithfulness, not citation accuracy.
The exception is factor discovery. Linking an event to causal indicators (FEDFUNDS, NY.GDP.MKTP.CD, V‑Dem codes) requires real dataset identifiers. Local Qwen 7B reliably hallucinates these; Claude reliably does not. So factor discovery is the one place where Claude is kept, and even there we persist metadata only — source, series ID, URL — never values. Reviewers can spot‑check IDs against the real databases.
The retrieval boundary is therefore drawn around what must be verifiable: paraphrased narrative goes local; cited references go to a model trained to cite, with a cost cap on top.
RAG is a tax. Pay it where outputs must be verifiable; skip it where outputs are stylistic. The right split is per‑task, not per‑system.
The eval pipeline samples eight entity types, runs deterministic heuristic checks (word counts, proper‑noun density, anchor lexical diversity, schema conformity, coverage floors), and optionally calls Claude with per‑entity rubrics. The judge is mandatory‑soft: nightly runs never break when the key is missing, but pre‑merge gates can force --judge required.
Cross‑architecture matters. A Qwen judge would inherit Qwen blind spots; Claude catches what Qwen produced and Qwen accepted. The historical‑example check is the only signal for fabricated history — no heuristic can tell a real 1987 Mexico debt crisis from a fabricated 1987 Tashkent Accords.
Every report includes a content‑addressed config snapshot (sha256 prefix). Pass --diff‑config A B and the diff replays. That made every change auditable without re‑running the pipeline.
Spend the API budget where it buys orthogonal signal. Same‑model judging is a confidence‑inflation machine.
Failures clustered into four families. Format failures were loud and easy to fix once we wrote tolerant parsers. Confidence fabrication was silent — the classifier emitted three values, but everything downstream treated them as a distribution. Semantic bugs in probability hid in defensive defaults: a binary 0¢ became 50%; exclusives clamped at 100%. And hallucinated history survived because no heuristic could distinguish a real debt crisis from a fabricated one.
Each fix was specific. Confidence: logit decoding gave 46 distinct softmax values. Probability: an explicit UNKNOWN path reclassified 570 events. History: the Claude judge added historical_example_realness as a 1–5 rubric.
Every failure mode was invisible until a check existed for it. Observability is the prerequisite for fixing — not the reward for fixing.
Defense in depth is ordered by cost. The first four layers are deterministic and run on every event. Layers 5–6 sample. Layer 7 (SummaryReport + BreaksReport with reason_counts) summarises the run for human eyes.
The most important guardrails are the cheapest. priority_score filters ~70% of events before any model is loaded. max_items caps absolute spend at any market size. Pydantic schemas + a 16‑source whitelist (FRED, World Bank, V‑Dem, ACLED, GDELT…) reject malformed factor indicators before they reach storage.
The expensive layer (Claude judge) only catches what the cheap ones missed: semantic faithfulness and hallucinated history.
Stack guardrails in increasing order of cost. Each layer earns its keep by catching errors the previous one could not see.
The cost curve has four points. Phase A: Claude does everything — $5–15/run, low latency, scales with the market universe. Phase B: a local summary path lands; scripts and prophecies stay on Claude — $2–5. Phase C (2026‑04‑29): full local cutover for generation; only the eval judge remains on Claude — ~$0.50. Phase D (post 2026‑05‑13): the judge expands from prophecies‑only to all artifacts, capped at judge_max_calls=600 — back up to $1–3.
The latency tradeoff is real. Local MLX 7B is slower per event than the API; but the job runs in a single process with no network round‑trips, so wall‑clock for the full pipeline is comparable.
The win is structural: cost no longer scales with the market universe. Doubling Kalshi's event count moves wall‑clock, not dollars.
Local generation eliminates per‑event API spend. Cost caps on the remaining Claude calls decouple absolute spend from upstream growth.
The pipeline runs as seven independently restartable stages, each owning its model load. A failed factors job does not redo summaries; an extra update-prophecies call is cheap.
Two operational moves stand out. Per‑config data folders (data/default/, data/tighter/) make A/B testing a one‑flag operation with zero data sharing — no more cp shuffles or cache‑TTL hacks. Dual deployment ships the same UI to either a FastAPI dev server or a static GitHub Pages bundle. A fetch interceptor decides at runtime which mode it is in — one source tree, two zero‑config deploys.
For day‑to‑day diagnosis, BreaksReport.reason_counts collapses a 1,772‑row gap list into a one‑line policy summary: how many events were dropped, by which gate, in which stage.
Stage isolation + per‑config data + dual‑mode deploy turned operations into config‑swap exercises, not code changes.
The discoveries cluster into two shapes. Three are about tooling intuition being wrong: prompt edits do not move small models; /no_think is silently ignored; logit decoding produces far richer confidence than expected.
Three are about interpreting eval signals: a falling "Other" recall is good (the garbage‑bin is gone); a 23.6% probability mismatch turns out to be measurement of real market edges; absolute cost caps beat threshold tuning under growth.
None of the six were predicted in advance. All six emerged from looking at a number the eval pipeline put on screen.
The most valuable findings were not designed‑in. They were surfaced by checks that already had to exist for other reasons.
The end‑state mirrors the principle that emerged from every retrospective decision. Local models do the work that tolerates paraphrase: summaries, scripts, prophecies, and the enrichment classifier (now logit‑decoded). Claude does the work where verifiability matters: causal factors with real series IDs, and cross‑architecture eval judging.
Everything persists as JSON under data/<config>/. The same source tree serves either a FastAPI dev server or a static GitHub Pages bundle via a fetch‑interceptor adapter. The eval pipeline writes pass‑rates and config snapshot IDs back into the store, closing the feedback loop.
What changed from the day‑one architecture (slide 3): the generation column moved from cloud to local; an eval column was added; data moved from a flat directory to per‑config worlds; the serving layer became dual‑mode.
The system did not get cleverer over five weeks. It got more aware of what each component was good at — and bounded each one accordingly.
Five lessons, each backed by a specific commit, a specific number, and a specific behavior change in the system. Together they describe a posture more than a methodology: treat every defensive default as a future bug, every fallback as a measurement opportunity, and every model boundary as a cost decision.
The lesson that ties all five together: structured observability beats ad‑hoc prompt tuning. Three of the five fixes (confidence, cost, UNKNOWN) only happened because the eval pipeline already existed to display the broken value.
Build the scaffolding before you need it. Then let it tell you what is wrong.
Structured observability beats ad‑hoc prompt tuning. Build the checks first, and the right fixes follow.