FE-1407: Catalogue elicitor failures that published measures miss - #9319
Queued
lunelson wants to merge 8 commits into
Queued
FE-1407: Catalogue elicitor failures that published measures miss#9319lunelson wants to merge 8 commits into
lunelson wants to merge 8 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
3 tasks
github-actions
Bot
added
area/infra
This was referenced Aug 25, 2026
3 tasks
Contributor
Author
This was referenced Aug 25, 2026
lunelson
changed the title
lunelson
marked this pull request as ready for review
August 25, 2026 12:13
PR SummaryLow Risk Overview Each mode is documented with operational class, labelled evidence, mechanism, a detection signature (spec for later oracles, not an implemented checker), primary accountable layer (disposition / technique / machinery), a graded prevention claim, and successor evidence pointers to FE-1402, FE-1403, FE-1404, FE-1406, and FE-1431. The doc also defines licensed deferral (durable deposit) versus evasive “later” promises, states explicit limits (no detectors, rates, or proof of prevention), and registers the file in Reviewed by Cursor Bugbot for commit a4dbcf3. Bugbot is set up for automated code reviews on this repo. Configure here. |
lunelson
force-pushed
the
ln/fe-1480-resolve-decision
branch
from
August 25, 2026 12:52
4fcb281 to
ec6cdec
Compare
lunelson
force-pushed
the
ln/fe-1407-catalogue-elicitor-failures
branch
from
August 25, 2026 12:52
9443650 to
0f619da
Compare
3 tasks
lunelson
force-pushed
the
ln/fe-1407-catalogue-elicitor-failures
branch
from
August 26, 2026 08:30
0f619da to
d256616
Compare
lunelson
force-pushed
the
ln/fe-1480-resolve-decision
branch
from
August 26, 2026 08:30
ec6cdec to
245487e
Compare
This was referenced Aug 26, 2026
lunelson
force-pushed
the
ln/fe-1407-catalogue-elicitor-failures
branch
from
August 26, 2026 08:50
d256616 to
6a57484
Compare
CiaranMn
previously approved these changes
Aug 26, 2026
Enable type-aware and compiler-backed linting across the Brunch packages, share scoped rules with the app, and resolve the initial diagnostics without widening repository policy. Co-authored-by: Cursor <cursoragent@cursor.com>
Derive persisted, transport, and test types from their schemas and library contracts so runtime boundaries and TypeScript consumers cannot drift independently. Co-authored-by: Cursor <cursoragent@cursor.com>
Record deterministic scaffolds and typed code obligations as the pure projection output, with executable TypeScript realized downstream through Petrinaut client tools. Reconcile Brunch steering, specs, glossary, and ADR history while leaving FE-1480 implementation priority open. Amp-Thread-ID: https://ampcode.com/threads/T-01a03444-b6bb-77de-b8ff-52509ae2d746 Co-authored-by: Amp <amp@ampcode.com>
Record ADR-0005's deterministic scaffold and code-obligation boundary while keeping projection and artifact realization honestly pending. Amp-Thread-ID: https://ampcode.com/threads/T-01a03444-b6bb-77de-b8ff-52509ae2d746 Co-authored-by: Amp <amp@ampcode.com>
Prioritize an evidence-backed non-HITL issue sequence before runtime implementation, then order the reviewer and semantic build paths. Record the proof bundle, build-readiness gates, and corrected Gherkin-before-CPS sequencing. Amp-Thread-ID: https://ampcode.com/threads/T-01a03444-b6bb-77de-b8ff-52509ae2d746 Co-authored-by: Amp <amp@ampcode.com>
Record the approved Linear blocker joins and distinguish FE-1431 design closure from later plugin ratification. Keep the remaining issue order strategic rather than turning the whole queue into a dependency graph. Amp-Thread-ID: https://ampcode.com/threads/T-01a03444-b6bb-77de-b8ff-52509ae2d746 Co-authored-by: Amp <amp@ampcode.com>
Treat ordered proof frontiers as epistemic strategies whose results reshape successor briefs. Require a fixed pre-commit review packet, independent legibility review, closed findings, and renewed review after remediation before the orchestrator authorizes commit.
Classify fifteen baseline and published failure modes by typed evidence, mechanism, detection signature, accountable layer, bounded prevention claim, and successor evidence input. Define licensed deferral as durable deposit rather than a promise, while preserving prompt-only and single-run limits. Independent specification and standards reviews passed after evidence typing, disjunctive FM-15 detection, completion-vocabulary restraint, authority-boundary repair, and removal of the non-procedural FE-1404 pointer from the completed baseline protocol. Retain FM-13 and granular successor inputs because they preserve the human-measure mismatch and the evidence needed for ordered successor briefs. Verification: 128 core tests passed; docs index 7/7 passed; git diff --check passed; Linear graph 84 issues/68 hard edges/no cycles/no assignee mismatches. FE-1404 Agent notes now explicitly score applicable catalogue signatures under the approved external-write gate.
lunelson
force-pushed
the
ln/fe-1407-catalogue-elicitor-failures
branch
from
August 26, 2026 10:42
6a57484 to
a4dbcf3
Compare
lunelson
force-pushed
the
ln/fe-1480-resolve-decision
branch
from
August 26, 2026 10:42
75ccc94 to
2224d62
Compare
github-actions
Bot
added
area/deps
kostandinang
approved these changes
Aug 26, 2026
lunelson
added this pull request to the merge queue
Aug 26, 2026
Any commits made after this event will not be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Brunch is an elicitation harness: an AI interviewer that draws a model of a cyber-physical process out of a human expert. The published measures we score interviews against (questionnaire quality, coverage, provenance) miss a class of failures that only show up when a frontier model does the interviewing. This pull request lands a catalogue of fifteen such failure modes, drawn from the two scored baseline runs and the literature, each typed by evidence, mechanism, detection signature, and accountable layer. Under the strategy pivot recorded later in this stack (S-007, ADR-0006), the catalogue's role is an oracle list: each signature is a check the walking-skeleton run is tested against.
🔗 Related links
libs/@hashintel/brunch-agent/docs/reference/research/elicitation/frontier-model-elicitor-failure-catalogue.mddocs/control/STEERING.md(Proof 1) and the FE-1404 issue, both in the top branch of this stack🚫 Blocked by
🔍 What does this change?
Adds one reference document and its index row. Fifteen failure modes (FM-01 to FM-15) are classified by the evidence that shows them, the mechanism behind them, a detection signature that a transcript or model can be tested for, the layer accountable for preventing them (interviewer, harness, plugin, or human measure), a bounded prevention claim, and the successor evidence each one needs. Licensed deferral is defined as a durable deposit rather than a promise. FM-13 is retained deliberately because it records where human measures and machine measures disagree.
🏗️ Agent notes
Stacked as
#9311→#9312→#9313→ this branch → #9320 → #9321 → #9322.Commit
944365009eCatalogue frontier-model elicitor failuresStatus under S-007
The design-convergence queue this issue opened (FE-1407 → FE-1402 → FE-1403 → FE-1404) was closed by strategy entry S-007 and ADR-0006 in the FE-1404 branch of this stack. The catalogue survives as test-bed material and as the oracle list for Proof 1; its former hard-blocker relation to FE-1404 was removed on 2026-08-25 because Linear blockers state current prerequisites, not history. The FE-1404 skeleton run records a verdict per applicable signature: observed, not observed, or not observable.
Verification at authoring
128 core tests, docs-index 7/7,
git diff --check, Linear graph audit (no cycles, no assignee mismatches). Re-verified aftergt syncat the stack tip: 182/182 core tests, 0 Oxlint errors.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
The catalogue's prose predates ADR-0006 and still calls
docs/specs/plugin-contract.md"provisional"; that remains true, so it was left unchanged.🐾 Next steps
FE-1404 (top of this stack, then a later run branch) tests the skeleton run against each signature.
🛡 What tests cover this?
packages/core/test/architecture/docs-index.test.ts— the INDEX row resolves and the document's relative links check.❓ How to test this?
docs/reference/research/elicitation/frontier-model-elicitor-failure-catalogue.md.npx vitest runinlibs/@hashintel/brunch-agent/packages/coreand confirm the docs-index gate passes.📹 Demo
Documentation only.