Skip to content

workiq: add retrieve reference doc and SKILL.md pointer - #191

Draft
Tolga Kilicli (tolgaki) wants to merge 1 commit into
mainfrom
tolgaki-microsoft-retrieve-skill-reference
Draft

workiq: add retrieve reference doc and SKILL.md pointer#191
Tolga Kilicli (tolgaki) wants to merge 1 commit into
mainfrom
tolgaki-microsoft-retrieve-skill-reference

Conversation

@tolgaki

@tolgaki Tolga Kilicli (tolgaki) commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

The gap

The WorkIQ skill ships a reference file for every MCP tool except one: retrieve.

That is the wrong one to be missing. retrieve is the default path for open-ended finding — the skill routes volume to it and then leaves its query quality completely unmanaged. Sixteen reference files, none for the tool that answers "where is the doc about X".

The failure mode

This was found empirically, not theoretically. An agent called retrieve with an unpunctuated keyword-mush query and got ten confident, well-formatted, completely wrong results — items from an unrelated workstream several years old — while the correct recent document went unfound.

There was no error. Nothing in the response signalled the failure. The results looked exactly like good results.

What varying the query shape showed

Six runs, same user intent, varying only the shape of the query string:

Query shape Terms actually extracted Outcome
Bare unpunctuated keyword string 2 generic nouns Largest candidate pool; 10 hits, none relevant
Full natural-language sentence 8 terms, entities preserved Target found at rank 1
Short comma-separated anchor groups 3 terms, all high-specificity Smallest candidate pool; best precision
Long query with extraction instructions and pasted figures 5 low-specificity terms Largest pool; target lost entirely

The pattern is consistent across all six: the more non-anchor tokens in the query, the more generic the extracted terms, the larger the candidate pool, and the worse the result.

The rule

retrieve rewards anchor density, not query length or politeness.

An anchor is a high-IDF term — a product name, person, metric, identifier, or date. Generic topic nouns (deck, update, latency, api, review) do not anchor; they match enormous numbers of items. Every non-anchor token dilutes the query.

Note that the tool schema's "natural-language queries" is a misleading proxy. Prose works only because writing a sentence forces you to name entities — rows 2 and 4 above show a longer, more natural query performing worse than a short comma-separated anchor group. Length is not the variable; anchors are.

The reference file also documents the free self-check: the response's SearchMetadata.searchTerms shows the terms actually used. If it lists only generic words, the query was diluted — reshape and retry rather than reporting the hits.

Changes

  • references/retrieve-work-iq.md (new, both workiq and workiq-preview) — parameters, the anchor-density rule, the observed anti-pattern table, the searchTerms self-check, when-to-use / do-NOT-use routing, the known chart-and-image recall limitation, and worked examples.
  • SKILL.md (both plugins) — a minimal ### retrieve section placed directly after ask in ## MCP Tools, carrying the anchor rule inline (SKILL.md is always in context; reference files load on demand) plus a references/retrieve-work-iq.md pointer matching the existing ask pointer pattern.

No manifest changes: reference files are not enumerated in marketplace.json, server.json, or any plugin manifest, and the repo has no docs linter.

Notes for reviewers

Deviations from the original plan, and why:

  1. No ## Content Safety section. The draft carried one with a [trust](../../trust/SKILL.md) link. That section and the trust skill exist only on workiq-domain-skill-hierarchy (Restructure workiq into a hub-and-domain skill hierarchy #182), not on main — no reference file on main has a Content Safety block, so including it would have introduced a broken link and a section with no local precedent. It should be added when Restructure workiq into a hub-and-domain skill hierarchy #182 lands.

  2. A short ### retrieve section rather than an edit to an existing bullet. The plan called for appending the anchor sentence to a retrieve is the default for open-ended finding hard-limits bullet. That bullet is also only on Restructure workiq into a hub-and-domain skill hierarchy #182retrieve appears zero times in main's SKILL.md. A pointer needs a heading to live under, so this adds the smallest section that gives it one.

  3. Routing tension with main. main's routing table still sends "finding documents by topic" and "who owns X" to ask, because it predates retrieve. The new reference states the retrieve-first position. Restructure workiq into a hub-and-domain skill hierarchy #182 aligns the table; until then the two disagree, and this PR does not restructure the table to fix it.

Related work: #163 ([DO NOT MERGE]) adds a different, longer retrieve-work-iq.md. That doc covers the copilot vs grounding strategy decision and the retrieve/ask boundary in much more depth; it does not cover query formulation. The two are complementary rather than competing — if #163 lands first, the anchor-density material here should be merged into it rather than kept as a second file.

Caveat

These observations come from one tenant and one session. They are directionally strong and internally consistent — the candidate-pool sizes and extracted search terms move together across all six runs — but this is not a controlled benchmark. Corpus composition, index size, and index state all plausibly affect the result.

Recommend adding a query-formulation axis to the retrieval evals: hold user intent fixed, vary query shape across bare keywords / prose / anchor groups, and measure precision and candidate-pool size. That would confirm the rule and quantify how much measured retrieval variance is attributable to query shape rather than retrieval quality.

`retrieve` was the only WorkIQ MCP tool without a reference file, despite
being the default path for open-ended finding. Its query quality was
therefore completely unmanaged.

Adds references/retrieve-work-iq.md to both the workiq and workiq-preview
skills, documenting the anchor-density rule, the observed anti-patterns,
and the SearchMetadata.searchTerms self-check. Adds a minimal `retrieve`
section to both SKILL.md files carrying the anchor rule inline plus the
reference pointer, matching the existing `ask` pointer pattern.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tolgaki
Tolga Kilicli (tolgaki) force-pushed the tolgaki-microsoft-retrieve-skill-reference branch from 652ab9a to fc5700c Compare August 29, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant