Skip to content

Calibrate copilot-instructions for non-Opus models - #132224

Open
tannergooding wants to merge 2 commits into
dotnet:mainfrom
tannergooding:tannergooding-calibrate-copilot-instructions
Open

Calibrate copilot-instructions for non-Opus models#132224
tannergooding wants to merge 2 commits into
dotnet:mainfrom
tannergooding:tannergooding-calibrate-copilot-instructions

Conversation

@tannergooding

Copy link
Copy Markdown
Member

Model auto-selection routes sessions across a range of models, and the weaker ones share a set of failure modes this file doesn't currently guard against: stopping mid-task for no blocking reason, reporting work as done that was never run, fusing a multi-part answer into one paragraph so the skipped parts are invisible, not volunteering a bug it noticed, and skipping a step because a tool is named differently than the instructions assume. Stronger models mostly avoid these on their own, so the new rules are inert for them and load-bearing for everyone else.

Four rules added, one or two lines each — Finish the task before you yield, Volunteer what you notice, Answer every question, and Tool and skill names are capabilities, not literals. The last matters most in this repo specifically: the file names seven skills, and "invoke a named skill rather than assuming what it says" is the difference between a model running build-and-test and guessing the build command.


Four existing rules had latent misreadings and are hardened in place:

  • Validation is now obligation-first. Scale the effort to the risk … say you didn't build and move on was readable as "tests are optional when you're confident," and the surviving rule required a fresh pass per edit. Now: one pass may cover several related edits if it exercises everything changed since the last one, and anything touching behavior, codegen, or a public contract gets the build and the relevant tests before you call it done. The comment/doc exemption is kept, with the caveat that a bad <see cref> or stray whitespace still fails a build.
  • Both push bullets. Always wait for explicit instruction to push plus wait for an explicit push instruction. This creates a mandatory decision point is the pair that makes a model commit and then stop to ask for permission it was already given by "open the PR". A request that entails a push now is that instruction — and it authorizes the push, not skipping validation or the target check. mandatory decision point also read as the direct inverse of not a checkpoint partway in.
  • Review feedback → any reported case, since an issue is a sample the same way a review comment is, plus name what you're leaving rather than quietly expanding scope into it.
  • Parallel tool calls — "independent" now excludes a call whose input comes from another's output.

No new sections and no restructuring.


Reviewed across three model families over two rounds, each tracing the same concrete task against the file rather than critiquing the prose. The push contradiction was found by all three, and one of them actually failed on it — committing and then stalling to ask. Final round returned no blocking findings from two of three; the dissent was a stall its own trace didn't reproduce.

Out of scope: trimming unrelated pre-existing rules is an editorial pass, not a behavioral one. One real ambiguity was left alone for the same reason — Under CCA, invoke it before making any code changes in Building & Testing can be read as requiring a full baseline build before a typo fix, which sits awkwardly next to the comment/doc exemption. Happy to file that separately.

Note

This PR description was drafted by GitHub Copilot.

Model auto-selection routes sessions to models weaker than Opus, which share
a set of failure modes this file doesn't guard against: stopping mid-task for
no blocking reason, reporting work as done that was never run, fusing a
multi-part answer into one paragraph, not volunteering a noticed bug, and
skipping a step because a tool is named differently than assumed.

Add four rules covering those, and harden four existing ones that were
readable as worse instructions -- most importantly the push pair, where
'wait for an explicit push instruction' made a model stop to ask for
permission it had already been given.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 12, 2026 17:47
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the repository’s Copilot behavioral guardrails to better support weaker auto-selected models, focusing on avoiding premature yielding, under-reporting, and tool/skill name brittleness.

Changes:

  • Tightens “Don’t claim more than you verified” and clarifies what constitutes sufficient validation coverage across related edits.
  • Adds four short behavioral rules: finish-before-yielding, volunteer observations, answer all parts, and treat tool/skill names as capabilities rather than literal strings.
  • Refines guidance around “reported case as sample” and push authorization, and clarifies that dependent tool calls shouldn’t be batched in parallel.
Show a summary per file
File Description
.github/copilot-instructions.md Calibrates Copilot instruction wording to reduce common failure modes in weaker models and hardens a few ambiguous existing rules.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread .github/copilot-instructions.md Outdated
An unread result means the call can't be counted as success, not that it
failed -- asserting the failure is itself a claim beyond what was verified.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 12, 2026 17:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants