Skip to content

Add analytics-opensearch-expertise skill - #50

Open
genedragon wants to merge 1 commit into
aws:mainfrom
genedragon:feature/analytics-opensearch-expertise
Open

Add analytics-opensearch-expertise skill#50
genedragon wants to merge 1 commit into
aws:mainfrom
genedragon:feature/analytics-opensearch-expertise

Conversation

@genedragon

Copy link
Copy Markdown

Description

Adds analytics-opensearch-expertise, a read-only health assessment skill for Amazon OpenSearch Service domains. Given a domain ARN, the skill guides the agent through 24 deterministic checks across 5 categories — cluster health, storage & shards, performance, security, and cost optimization — using only control-plane APIs (es:Describe*/es:List*, covered by the DevOps Agent base policy) and CloudWatch metrics.

Key design points:

  • Exact severity logic per check (thresholds, statistics, and tier rules are pinned in the spec, not left to agent judgment) with a defined category roll-up rule
  • Verbatim remediation dictionary (24 entries with official developer-guide links) in references/, loaded via read_skill_resource
  • API compatibility guidance for the agent's legacy Elasticsearch-compatible client, including a CloudWatch metrics fallback for shard-level health so checks degrade gracefully instead of being skipped
  • Explicit SKIPPED paths for missing datapoints and permission denials — the agent reports gaps rather than inventing results
  • Works with both the Chat and Investigations (incident RCA) subagents

Type of change

  • New skill

Testing

Agent Skill Eval (results included in evals/):

  • report.json root "passed": true — overall 0.88 (B)
  • Audit: 96/100 (A), zero critical/warning findings
  • Trigger reliability: 1.00 (A) — all positive and negative activation queries correct
  • Functional: passed (outcome/style/efficiency 1.0)
  • 14/14 golden eval cases pass

Manual DevOps Agent testing (dedicated test account, live OpenSearch domains):

  • 10+ assessment runs across three domain configurations (healthy baseline, security-features-off, and a seeded degraded domain with sustained YELLOW status and high shard density)
  • Skill activation confirmed in reasoning traces using realistic prompts that never name the skill
  • Tested in both Chat and Investigations subagents; an incident RCA investigation correctly identified a seeded root cause (over-replication on a 3-node single-AZ cluster) using the skill's API-compatibility and fallback guidance
  • Repeated runs to validate consistency; findings fed iterative spec refinements (v1.0 → v2.6, see CHANGELOG.md)

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant