Deterministic self-healing drift: remove the LLM from remediation#335
Merged
Conversation
… poison the batch
…tion to deterministic sync The fix-drift LLM-in-the-loop remediation path (scripts/fix-drift.ts) and its anti-cheat success predicate are removed. Remediation for drift now runs through the deterministic model-family sync (drift-sync.ts / drift-sync-check.ts introduced in the prior commit): known model-family changes are applied programmatically, anything outside that scope is flagged with a needs-human note, and a human still reviews and merges. The fix-drift workflow, Slack summary, and shared drift-types are updated to call the new sync path instead of invoking an LLM to freewrite a fix.
commit: |
…re cases
The `drift-live-pr` CI check quarantined the whole batch as an unparseable
exit-5 failure:
expected 'claude-haiku-4-5-20251001' to be 'claude-haiku-5-1-20260201'
Root cause is cross-block pollution of `resolveLiveModel`'s per-key memo, not
a model-pin mismatch. Both sides of the live-leg 3-way comparison already use
the same dynamically-resolved id. But when ANTHROPIC_API_KEY is armed (CI),
the live `describe.skipIf` block runs FIRST, resolves the real still-live id,
and memoizes it under the shared "anthropic" key — with no afterEach reset,
exactly like the real live block. The fixture self-healing block reset the
memo only in afterEach, so its FIRST case inherited the leaked live id instead
of resolving against its own stub, then asserted a raw `.toBe()` on model ids.
That AssertionError is not a structured drift report, so the collector could
not parse it (exit 5, fatal quarantine) rather than a non-fatal exit-2 report.
Fix: reset the memo in `beforeEach` of the fixture block so each case resolves
against its own stub regardless of what the live block left cached. Add a
seeding regression guard that simulates the armed-key live block populating the
memo first (no credentials needed), reproducing the CI ordering deterministically.
Siblings audited: ws-gemini-live isolates via a distinct cache key
(gemini-live-text-unittest); openai-responses/ws-responses have no fixture
block reading the memo after their live block; openai-chat has no such
assertion. anthropic was the only leg sharing one memo key across both blocks.
Local red-green (fixture-driven; live keys unavailable locally, so the true
confirmation is the re-run drift-live-pr CI):
RED (before fix): REGRESSION case fails with the verbatim CI assertion.
GREEN (after fix): all 11 runnable anthropic cases pass (6 live-key skipped).
Full gate: tsc 0, eslint/prettier clean, unit 4722 passed (logic-pin 14,
fix-drift-workflow 37), drift 118 passed, tsdown build OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deterministic self-healing drift — remove the LLM from remediation
The drift auto-remediation subsystem broke ~8 times because it let an LLM freewrite edits to the detector's own fixtures and then tried to police that with a 916-line anti-cheat predicate — an unwinnable loop. This replaces that whole loop with robustness-by-construction: most drift now self-heals with no fix at all, mechanical model-family churn is synced deterministically, and the human surface shrinks to exactly two real decisions (mock a brand-new family or not; follow a genuine protocol redesign or not).
What changed
scripts/fix-drift.ts(the freewriter) andscripts/drift-success-predicate.ts(the 916-line predicate) are deleted, along with their now-orphaned tests./modelsand either:ok-applied— mechanically editsmodel-registry.ts(via the TypeScript compiler API, not text munging) to drop a deprecated family → opens a PR.needs-human— a brand-new/unclassified family, a still-referenced deprecation, or a registry structural mismatch → writes adrift-proposals/note and opens a PR where a human setsDecision: include; the next run applies it.scripts/drift-sync-check.ts(changed-file allowlist + classification checksum-pin re-assert + clean re-collect, all fail-closed) and are idempotent across daily re-fires (deduped on a stable content-hash changeset key). No auto-merge — humans merge.resolveLiveModel/isInfraSkiphelpers.Review
Built as 14 parallel slots, then a Tier-3 review that found and fixed 8 mandatory defects — each with a local red-green proof, each re-reviewed to zero on the fixed state. Notably: the sync core's recollect gate was silently breaking the new-family human-routing path (and a fake test was hiding it), and the workflow didn't persist the needs-human note (so the human-decision path was unreachable). Both would have shipped green otherwise.
Test evidence
tsc --noEmit0 errors · full suite 4766 passed / 0 failed · build ok · classification pin 14/14 · workflow tests 37/37.Fast-follow (non-blocking, not in this PR)
3 now-inert helper functions to delete, a couple of dead workflow permissions,
excludeFamiliesdata not yet pinned, the byte-mirror not equivalence-tested,claude-fable-5re-proposed each run, and a pre-existing Bedrock$metadatashape gap (only surfaces under real AWS creds).🤖 Generated with Claude Code
https://claude.ai/code/session_01KamK73Wu5heLxJEogM6hHC