Skip to content

feat(skills): validation and shadowing signals#2607

Merged
k11kirky merged 2 commits into
mainfrom
posthog-code/skills-04-validation
Jun 12, 2026
Merged

feat(skills): validation and shadowing signals#2607
k11kirky merged 2 commits into
mainfrom
posthog-code/skills-04-validation

Conversation

@k11kirky

@k11kirky k11kirky commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Skill configurations can have subtle issues — missing descriptions, frontmatter names that don't match their directory, oversized SKILL.md files, or skills being silently shadowed by higher-priority sources — with no feedback to the user. This adds health analysis so these problems are surfaced directly in the Skills UI.

Changes

  • Added analyzeSkills, a pure function that inspects a list of SkillInfo objects and returns a map of issues keyed by skill path. It detects four issue types:
    • missing-description — agents use the description to decide when to invoke a skill
    • name-mismatch — frontmatter name differs from the directory name
    • oversized-manifestSKILL.md exceeds 32 kB, adding context cost on every use
    • shadowed — a higher-precedence source (repo > user > marketplace > bundled) provides a skill with the same name
  • Added skillMdBytes to SkillInfo (and its Zod schema) so the file size is available without additional I/O; the discovery layer populates it from the raw file content.
  • SkillsView runs analyzeSkills over the loaded skills list and passes the resulting SkillAnalysis down to SkillSection and SkillDetailPanel.
  • SkillCard shows a warning icon with a tooltip listing all issue messages when a skill has issues.
  • SkillDetailPanel renders each issue as an amber Callout below the skill header.

How did you test this?

  • Added unit tests for analyzeSkills covering: healthy skills, missing/whitespace descriptions, name mismatches, oversized manifests, all shadowing precedence pairs, three-way collisions, and multiple issues accumulating on a single skill.
  • Updated the existing readSkillMetadataFromDir test to include the new skillMdBytes field in the expected output.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown

React Doctor could not complete this scan.

No React dependency found in /tmp/react-doctor-baseline-rFJRO2/package.json. Add "react" to dependencies (or peerDependencies) and re-run.

Reviewed by React Doctor for commit 108c18e.

@greptile-apps

greptile-apps Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
packages/ui/src/features/skills/SkillCard.tsx:78-80
When a skill has multiple issues the tooltip messages are joined with `
`, but HTML collapses whitespace in text nodes, so all messages run together without any visible separator. Radix UI Tooltip renders `content` as a `ReactNode`, so newlines in a plain string don't produce line breaks.

```suggestion
        <Tooltip
          content={
            <Flex direction="column" gap="1">
              {issues.map((issue) => (
                <Text key={issue.type} size="1">
                  {issue.message}
                </Text>
              ))}
            </Flex>
          }
        >
          <Warning size={14} className="shrink-0 text-amber-11" />
        </Tooltip>
```

Reviews (1): Last reviewed commit: "feat(skills): validation and shadowing s..." | Re-trigger Greptile

Comment thread packages/ui/src/features/skills/SkillCard.tsx Outdated
@k11kirky k11kirky marked this pull request as ready for review June 11, 2026 13:54
Comment thread packages/core/src/skills/analyzeSkills.ts
@k11kirky k11kirky force-pushed the posthog-code/skills-03-live-refresh branch from a0b806d to 1e80286 Compare June 12, 2026 07:50
@k11kirky k11kirky force-pushed the posthog-code/skills-04-validation branch from 41afe29 to 0bdd245 Compare June 12, 2026 07:50
@k11kirky k11kirky force-pushed the posthog-code/skills-03-live-refresh branch from 1e80286 to 5df364a Compare June 12, 2026 08:19
@k11kirky k11kirky force-pushed the posthog-code/skills-04-validation branch from 0bdd245 to 17f53b7 Compare June 12, 2026 08:19

k11kirky commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

Merge activity

@k11kirky k11kirky force-pushed the posthog-code/skills-04-validation branch from 17f53b7 to 17c16d7 Compare June 12, 2026 11:43
@k11kirky k11kirky force-pushed the posthog-code/skills-03-live-refresh branch from 5df364a to 017c36e Compare June 12, 2026 11:43
@k11kirky k11kirky changed the base branch from posthog-code/skills-03-live-refresh to graphite-base/2607 June 12, 2026 11:55
@k11kirky k11kirky changed the base branch from graphite-base/2607 to main June 12, 2026 12:06
k11kirky added 2 commits June 12, 2026 12:06
- analyzeSkills in @posthog/core: pure, no-I/O health analysis over the
  discovered skills. Rules: missing/empty description, frontmatter name
  vs directory name mismatch, oversized SKILL.md (context-cost warning,
  32 kB threshold), and cross-source name collisions with explicit
  precedence (repo > user > marketplace > bundled) — losers are marked
  shadowed by the winner.
- SkillInfo gains skillMdBytes (computed during discovery) to feed the
  size rule.
- UI: amber health badge with tooltip on SkillCard; per-issue callouts
  in the detail panel ("Shadowed by the repository skill named ...").
- Unit tests cover every rule, the precedence matrix, the boundary
  size, and multi-issue accumulation.

Generated-By: PostHog Code
Task-Id: f4e84f1a-19c9-490c-9b98-47787a7dddcf
… when shadowed

Review feedback on #2607: newline-joined tooltip text collapses in HTML,
so issues render as separate lines now; shadow messages include the
winning repo's name so same-name skills across two open repos are
distinguishable.

Generated-By: PostHog Code
Task-Id: f4e84f1a-19c9-490c-9b98-47787a7dddcf
@k11kirky k11kirky force-pushed the posthog-code/skills-04-validation branch from 17c16d7 to 108c18e Compare June 12, 2026 12:06
@k11kirky k11kirky merged commit 37b31ad into main Jun 12, 2026
24 checks passed
@k11kirky k11kirky deleted the posthog-code/skills-04-validation branch June 12, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants