`transform_decisions` records the fold reason but not the hash values — "did the bytes actually change?" needs a source read instead of a query

[iceteaSA]

Summary

transform_decisions stores materialize_reason (system_hash, ttl_idle, first_render, …) but not the hash values behind it. That's enough to know which rule fired and not enough to know what changed.

Proposal: add system_hash_prev and system_hash_new to the row. Two columns on a table already written once per pass.

What made this concrete

I spent tonight partitioning a day's worth of full prefix busts on a live box — 27 of them, ~6.3M cache-write tokens at the raw ceiling — into prefix-changed / expired / evicted. MC's fold markers were the input: a bust matching a fold is explained, a bust with no fold is an anomaly. That worked, and the residue came out at 8 busts / 3,110,842 tokens.

Then a reviewer raised a fair challenge: for two system_hash folds with no identifiable cause, were the bytes actually different — or was MC re-materializing against a prefix that had already been dropped provider-side, making the fold a consequence rather than the cause? If the latter, those two belonged in the residue and the partition was wrong by 25%.

That question is not answerable from the table. There is no hash column, so there is no query that distinguishes "the stored hash differed from the computed one" from "there was no usable stored hash." I answered it by reading inject-compartments.ts instead:

// line 1517 — evaluated FIRST
if (!args.state.cachedM0Bytes) return { value: true, reason: "first_render" };
...
// line 1555 — only reachable when a cached anchor exists
if (hard.systemHash !== "" && hard.systemHash !== (args.state.cachedM0SystemHash ?? "")) {
    return { value: true, reason: "system_hash" };
}

The first_render short-circuit 38 lines earlier means system_hash can only fire when a stored hash existed and genuinely differed — so the challenge is refuted and the partition holds. But that took a source read, and it only works for someone who can read the source. The data alone couldn't answer it.

Worth noting both reasons are live in production, so this isn't a theoretical distinction: system_hash 578 uses, first_render 107, ttl_idle 116, pressure_refold 145 (my box, current DB).

Why the columns are worth more than they look

Diagnosis often has no external cross-check. In my case the provider exposes no cache-miss reason on the responses we were looking at, so MC's own markers were the only authoritative statement of why a prefix changed. When that's the situation, a marker that records the verdict without the evidence is doing half the job.

It makes the question retroactive. Tonight's investigation could only reason forward from code. With the values stored, the same question becomes SELECT ... WHERE system_hash_prev != system_hash_new over history — including for incidents nobody was watching live, which is when this normally gets asked.

It measures magnitude, not just occurrence. Two hashes let you see whether the prefix changed once or is churning per-pass. A reason string can't distinguish a one-off from a loop.

Shape

ALTER TABLE transform_decisions ADD COLUMN system_hash_prev TEXT;
ALTER TABLE transform_decisions ADD COLUMN system_hash_new  TEXT;

Nullable, no backfill — historical rows keep NULL and that reads correctly as "not recorded." Populated only when materialized = 1, since the values are meaningless otherwise. Both are already in hand at the write site (hard.systemHash and state.cachedM0SystemHash are the two operands of the comparison that sets the reason).

Storage is negligible — the hashes are short and the table is already per-pass.

Scope question for you

I've only argued the system_hash case because that's what I hit. The same gap plausibly exists for the other value-comparison reasons — model_change has cachedM0ModelKey vs the live key, project_memory_epoch has two epoch numbers. If you'd rather have one generic pair of columns (compared_prev / compared_new) covering every comparison-driven reason, that seems better than three special-cased pairs, and I'd defer to your preference on shape.

Happy to send the PR either way — migration, write-site, tests. Say which shape you want.