workiq: add retrieve reference doc and SKILL.md pointer - #191
Draft
Tolga Kilicli (tolgaki) wants to merge 1 commit into
Draft
workiq: add retrieve reference doc and SKILL.md pointer#191Tolga Kilicli (tolgaki) wants to merge 1 commit into
Tolga Kilicli (tolgaki) wants to merge 1 commit into
Conversation
`retrieve` was the only WorkIQ MCP tool without a reference file, despite being the default path for open-ended finding. Its query quality was therefore completely unmanaged. Adds references/retrieve-work-iq.md to both the workiq and workiq-preview skills, documenting the anchor-density rule, the observed anti-patterns, and the SearchMetadata.searchTerms self-check. Adds a minimal `retrieve` section to both SKILL.md files carrying the anchor rule inline plus the reference pointer, matching the existing `ask` pointer pattern. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tolga Kilicli (tolgaki)
force-pushed
the
tolgaki-microsoft-retrieve-skill-reference
branch
from
August 29, 2026 00:21
652ab9a to
fc5700c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
The WorkIQ skill ships a reference file for every MCP tool except one:
retrieve.That is the wrong one to be missing.
retrieveis the default path for open-ended finding — the skill routes volume to it and then leaves its query quality completely unmanaged. Sixteen reference files, none for the tool that answers "where is the doc about X".The failure mode
This was found empirically, not theoretically. An agent called
retrievewith an unpunctuated keyword-mush query and got ten confident, well-formatted, completely wrong results — items from an unrelated workstream several years old — while the correct recent document went unfound.There was no error. Nothing in the response signalled the failure. The results looked exactly like good results.
What varying the query shape showed
Six runs, same user intent, varying only the shape of the query string:
The pattern is consistent across all six: the more non-anchor tokens in the query, the more generic the extracted terms, the larger the candidate pool, and the worse the result.
The rule
retrieverewards anchor density, not query length or politeness.An anchor is a high-IDF term — a product name, person, metric, identifier, or date. Generic topic nouns (
deck,update,latency,api,review) do not anchor; they match enormous numbers of items. Every non-anchor token dilutes the query.Note that the tool schema's "natural-language queries" is a misleading proxy. Prose works only because writing a sentence forces you to name entities — rows 2 and 4 above show a longer, more natural query performing worse than a short comma-separated anchor group. Length is not the variable; anchors are.
The reference file also documents the free self-check: the response's
SearchMetadata.searchTermsshows the terms actually used. If it lists only generic words, the query was diluted — reshape and retry rather than reporting the hits.Changes
references/retrieve-work-iq.md(new, bothworkiqandworkiq-preview) — parameters, the anchor-density rule, the observed anti-pattern table, thesearchTermsself-check, when-to-use / do-NOT-use routing, the known chart-and-image recall limitation, and worked examples.SKILL.md(both plugins) — a minimal### retrievesection placed directly afteraskin## MCP Tools, carrying the anchor rule inline (SKILL.md is always in context; reference files load on demand) plus areferences/retrieve-work-iq.mdpointer matching the existingaskpointer pattern.No manifest changes: reference files are not enumerated in
marketplace.json,server.json, or any plugin manifest, and the repo has no docs linter.Notes for reviewers
Deviations from the original plan, and why:
No
## Content Safetysection. The draft carried one with a[trust](../../trust/SKILL.md)link. That section and thetrustskill exist only onworkiq-domain-skill-hierarchy(Restructure workiq into a hub-and-domain skill hierarchy #182), not onmain— no reference file onmainhas a Content Safety block, so including it would have introduced a broken link and a section with no local precedent. It should be added when Restructure workiq into a hub-and-domain skill hierarchy #182 lands.A short
### retrievesection rather than an edit to an existing bullet. The plan called for appending the anchor sentence to aretrieve is the default for open-ended findinghard-limits bullet. That bullet is also only on Restructure workiq into a hub-and-domain skill hierarchy #182 —retrieveappears zero times inmain'sSKILL.md. A pointer needs a heading to live under, so this adds the smallest section that gives it one.Routing tension with
main.main's routing table still sends "finding documents by topic" and "who owns X" toask, because it predatesretrieve. The new reference states theretrieve-first position. Restructure workiq into a hub-and-domain skill hierarchy #182 aligns the table; until then the two disagree, and this PR does not restructure the table to fix it.Related work: #163 (
[DO NOT MERGE]) adds a different, longerretrieve-work-iq.md. That doc covers thecopilotvsgroundingstrategy decision and theretrieve/askboundary in much more depth; it does not cover query formulation. The two are complementary rather than competing — if #163 lands first, the anchor-density material here should be merged into it rather than kept as a second file.Caveat
These observations come from one tenant and one session. They are directionally strong and internally consistent — the candidate-pool sizes and extracted search terms move together across all six runs — but this is not a controlled benchmark. Corpus composition, index size, and index state all plausibly affect the result.
Recommend adding a query-formulation axis to the retrieval evals: hold user intent fixed, vary query shape across bare keywords / prose / anchor groups, and measure precision and candidate-pool size. That would confirm the rule and quantify how much measured retrieval variance is attributable to query shape rather than retrieval quality.