Calibrate copilot-instructions for non-Opus models - #132224
Open
tannergooding wants to merge 2 commits into
Open
Calibrate copilot-instructions for non-Opus models#132224tannergooding wants to merge 2 commits into
tannergooding wants to merge 2 commits into
Conversation
Model auto-selection routes sessions to models weaker than Opus, which share a set of failure modes this file doesn't guard against: stopping mid-task for no blocking reason, reporting work as done that was never run, fusing a multi-part answer into one paragraph, not volunteering a noticed bug, and skipping a step because a tool is named differently than assumed. Add four rules covering those, and harden four existing ones that were readable as worse instructions -- most importantly the push pair, where 'wait for an explicit push instruction' made a model stop to ask for permission it had already been given. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 15 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the repository’s Copilot behavioral guardrails to better support weaker auto-selected models, focusing on avoiding premature yielding, under-reporting, and tool/skill name brittleness.
Changes:
- Tightens “Don’t claim more than you verified” and clarifies what constitutes sufficient validation coverage across related edits.
- Adds four short behavioral rules: finish-before-yielding, volunteer observations, answer all parts, and treat tool/skill names as capabilities rather than literal strings.
- Refines guidance around “reported case as sample” and push authorization, and clarifies that dependent tool calls shouldn’t be batched in parallel.
Show a summary per file
| File | Description |
|---|---|
| .github/copilot-instructions.md | Calibrates Copilot instruction wording to reduce common failure modes in weaker models and hardens a few ambiguous existing rules. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
An unread result means the call can't be counted as success, not that it failed -- asserting the failure is itself a claim beyond what was verified. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Model
auto-selection routes sessions across a range of models, and the weaker ones share a set of failure modes this file doesn't currently guard against: stopping mid-task for no blocking reason, reporting work as done that was never run, fusing a multi-part answer into one paragraph so the skipped parts are invisible, not volunteering a bug it noticed, and skipping a step because a tool is named differently than the instructions assume. Stronger models mostly avoid these on their own, so the new rules are inert for them and load-bearing for everyone else.Four rules added, one or two lines each — Finish the task before you yield, Volunteer what you notice, Answer every question, and Tool and skill names are capabilities, not literals. The last matters most in this repo specifically: the file names seven skills, and "invoke a named skill rather than assuming what it says" is the difference between a model running
build-and-testand guessing the build command.Four existing rules had latent misreadings and are hardened in place:
Scale the effort to the risk … say you didn't build and move onwas readable as "tests are optional when you're confident," and the surviving rule required a fresh pass per edit. Now: one pass may cover several related edits if it exercises everything changed since the last one, and anything touching behavior, codegen, or a public contract gets the build and the relevant tests before you call it done. The comment/doc exemption is kept, with the caveat that a bad<see cref>or stray whitespace still fails a build.Always wait for explicit instruction to pushpluswait for an explicit push instruction. This creates a mandatory decision pointis the pair that makes a model commit and then stop to ask for permission it was already given by "open the PR". A request that entails a push now is that instruction — and it authorizes the push, not skipping validation or the target check.mandatory decision pointalso read as the direct inverse ofnot a checkpoint partway in.No new sections and no restructuring.
Reviewed across three model families over two rounds, each tracing the same concrete task against the file rather than critiquing the prose. The push contradiction was found by all three, and one of them actually failed on it — committing and then stalling to ask. Final round returned no blocking findings from two of three; the dissent was a stall its own trace didn't reproduce.
Out of scope: trimming unrelated pre-existing rules is an editorial pass, not a behavioral one. One real ambiguity was left alone for the same reason —
Under CCA, invoke it before making any code changesin Building & Testing can be read as requiring a full baseline build before a typo fix, which sits awkwardly next to the comment/doc exemption. Happy to file that separately.Note
This PR description was drafted by GitHub Copilot.