Skip to content

Add EKS Node Diagnostics MCP server - #41

Open
shyamkulkarni wants to merge 10 commits into
aws:mainfrom
shyamkulkarni:eks-mcp-collection-approval
Open

Add EKS Node Diagnostics MCP server#41
shyamkulkarni wants to merge 10 commits into
aws:mainfrom
shyamkulkarni:eks-mcp-collection-approval

Conversation

@shyamkulkarni

Copy link
Copy Markdown

Description

Adds the AWS EKS Node Diagnostics MCP server (mcp/aws-eks-node-diagnostics-mcp): a CDK-deployed MCP server that lets the DevOps Agent collect and analyze EKS worker node logs via SSM Automation (the AWS-managed AWSSupport-CollectEKSInstanceLogs document), with 19 tools across four tiers (core operations, log analysis, cluster-level diagnostics, and 41 structured SOP runbooks).

Key design points:

  • Human-in-the-loop collection approval (on by default): the only mutating tools, collect and batch_collect, run through wrapper SSM Automation documents whose first step is the native aws:approve action. The execution pauses in SSM until a designated approver (IAM principals fixed at deploy time via APPROVAL_APPROVER_ARNS) approves in the Systems Manager console. Decisions are IAM-authenticated and CloudTrail-audited; the MCP Lambda has no ssm:SendAutomationSignal permission, so an agent cannot approve its own request. The agent just polls status() and collection proceeds automatically on approval.
  • Fail-closed security defaults: cluster allowlist required at synth time, region restriction, SSM document restriction, per-tool client ACLs, per-caller rate limiting, presigned URL expiry caps, and response redaction.
  • Auth and transport: Cognito OAuth2 in front of a BedrockAgentCore Gateway MCP endpoint; optional VPC deployment with S3/KMS endpoints.

Type of change

  • New skill
  • New custom agent
  • Update to an existing skill or agent
  • Documentation or infrastructure change (new MCP server)

Testing

  • npm run build (tsc) — passes
  • npm test — 12/12 jest tests passed (fast-check property tests for construct IAM and KMS/S3 policies), 44/44 pytest tests passed (approval workflow, region validation, tool validation wiring)
  • Manually deployed via deploy.sh and exercised the approval flow end to end: collect returns pending_approval with an SSM console deep link, approval in the console resumes collection, and status() reports the approval state and collection progress

License confirmation

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

Port the sample-eks-node-diagnostics-mcp project into
mcp/aws-eks-node-diagnostics-mcp/, including CDK infrastructure,
Lambda sources, diagnostic runbooks, CNI semantics, the packaged
skill, and tests.

The skill archive (skills/eks-node-diagnostics.zip) is force-added
because the repo's root .gitignore (*.zip) would otherwise exclude
it.
Drop skills/eks-node-diagnostics.zip and its README section per review
feedback; the skill is not required for the MCP server to function.
…proval

Remove the restricted tcpdump_capture/tcpdump_analyze tools and
gate the two mutating tools (collect, batch_collect) behind an
out-of-band human approval flow instead:

- Approval Lambda + Function URL with capability-URL tokens
  (SHA-256 stored server-side, single-use, DynamoDB TTL expiry)
- SNS topic notifies approvers with approve/deny links
- batch_collect defaults to dry-run
- Stricter EKS instance validation: user-settable
  kubernetes.io/cluster/* tag rejected unless
  ALLOW_SELF_MANAGED_NODES=true
- ReDoS-safe search regex validation and log-key path checks
- Update README, architecture docs, and runbooks accordingly
- Add pytest coverage for the approval flow
Replace the custom DynamoDB + Lambda Function URL approval flow with
SSM Automation wrapper documents whose first step is the native
aws:approve action. Approvers are fixed IAM principals set at deploy
time (APPROVAL_APPROVER_ARNS) and approve/deny in the Systems Manager
console; decisions are IAM-authenticated and CloudTrail-audited. The
agent polls status() and collection proceeds automatically on
approval, so no approvalId re-call is needed.
…proval

# Conflicts:
#	mcp/aws-eks-node-diagnostics-mcp/README.md
#	mcp/aws-eks-node-diagnostics-mcp/bin/app.ts
#	mcp/aws-eks-node-diagnostics-mcp/deploy.sh
#	mcp/aws-eks-node-diagnostics-mcp/src/lambda/ssm-automation-enhanced.py
#	mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-construct-v2.ts
#	mcp/aws-eks-node-diagnostics-mcp/src/ssm-automation-gateway-stack-v2.ts
#	mcp/aws-eks-node-diagnostics-mcp/tests/construct-iam.property.test.ts
#	mcp/aws-eks-node-diagnostics-mcp/tests/construct-kms-s3.property.test.ts
#	mcp/aws-eks-node-diagnostics-mcp/tests/test_collection_approval.py
@LearningNewbie
LearningNewbie self-requested a review August 14, 2026 15:16

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpicking, not a blocker: this JSDoc block (lines 2267–2278) describes the old DynamoDB/Function-URL approval mechanism (approvalId, token, DynamoDB record, approvalViaPublicUrl) that this exact PR removes. Looks like it got orphaned above getUnzipLambdaCode() during the refactor to native aws:approve — the function body below it is correct and unrelated to approval logic, just the comment is stale. Suggest deleting the second JSDoc block so it doesn't mislead the next reader into thinking there's a DynamoDB-based approval path elsewhere in the file.

Address PR review comments 2 and 3:

- Idempotent replay of collect() with a token whose execution is an
  approval wrapper now runs the same augment_wrapper_status enrichment
  as status(), so agents see humanApproval/consoleUrl instead of a
  bare InProgress while paused at aws:approve. The replay response
  also reports the execution's actual region.
- Add a synth-time CDK warning when requireCollectionApproval is on
  and allowedRegions spans multiple regions, since the approval
  wrapper documents exist only in the stack's home region.
- Add regression tests for both replay paths.
@shyamkulkarni

Copy link
Copy Markdown
Author

Addressed the review feedback in 44c60b8:

Comment 2 (idempotent replay loses approval context) — fixed. find_execution_by_idempotency_token now returns the execution's documentName, resolved region, and full execution snapshot, and the idempotent-return branch in start_log_collection runs the same augment_wrapper_status enrichment that status() uses whenever the existing execution is an approval wrapper. A retried collect() with the same token now reports humanApproval.state: pending with the SSM console deep link and polling guidance instead of a bare InProgress. The replay response also reports the execution's actual region rather than the current request's resolved region. Two regression tests cover the wrapper and plain-collection replay paths.

Comment 3 (single-region approval gate) — warning added. Single-region approval-gated collection is intentional for now: the wrapper documents are regional SSM documents owned by the stack, and embedding deployment-owned approvers/role/topic per region means the right fix is a stack per region rather than cross-region document sharing. Per the suggestion, synthesis now emits a cdk.Annotations warning when requireCollectionApproval is enabled and allowedRegions spans multiple regions, so operators learn at deploy time that approval-gated collect()/batch_collect() only works in the stack's home region.

Comment 4 (IAM document scoping) — agreed, deferring as a follow-up. Two corrections worth noting for whoever picks it up: (1) the scoped document list must also include the tcpdump approval wrapper, otherwise approval-gated captures break when the restricted tools are enabled; (2) the automation-definition/* resources in the same statements need the same scoping (with version qualifiers, e.g. automation-definition/<name>:$DEFAULT), otherwise the tightening is cosmetic since that resource type still authorizes StartAutomationExecution. The per-region lists also differ: wrapper documents exist only in the home region, while AWSSupport-CollectEKSInstanceLogs is AWS-owned in every allowed region.

Validation: 46 Python tests pass (including the 2 new regression tests) and the TypeScript build is clean. The pre-existing jest property-test failures (test-file parse errors and the missing lib/sops/runbooks asset) reproduce identically on the unmodified head and are unrelated; happy to fix those separately.

Address the MCP security review (Way In / Way Out threat model):

- M3: tcpdump_capture is now both opt-in (ENABLED_RESTRICTED_TOOLS)
  and gated by its own native SSM aws:approve wrapper document, so a
  poisoned agent cannot self-approve via the confirmCapture parameter.
- E1: kubernetes.io/cluster/* tags are no longer trusted alone. Only
  EKS-managed eks:cluster-name/eks:nodegroup-name tags are accepted,
  verified against the EKS API; self-managed nodes require explicit
  ALLOW_SELF_MANAGED_NODES opt-in with EKS API cross-check.
- E2: Lambda-level ALLOWED_CLUSTER_NAMES allowlist enforced on
  instance validation, cluster_health, and batch_collect.
- E4: validate_log_key enforces bundle-shaped keys, blocks traversal,
  and scopes reads to the instance under investigation.
- E5: catastrophic regex rejection plus SIGALRM wall-clock timeouts
  around search matching.
- Pod-scoped capture resolves PID/network namespace immediately
  before nsenter and rejects host-netns or changed namespaces.
- Runbooks updated for the approval-gated capture flow; tcpdump
  approval regression tests added.
Close the two residual gaps from the MCP security review:

- E2: an empty ALLOWED_CLUSTER_NAMES no longer permits all clusters by
  default. The Lambda now requires the deploy-time any-cluster
  acknowledgment (allowAnyClusterName -> ALLOW_ANY_CLUSTER_NAME) to
  operate without an allowlist, matching the CDK synth-time contract.
- E4: read() and artifact() now require instanceId, so every log read
  and presigned URL is scoped to the instance under investigation.
  Previously the cross-instance check only applied when the caller
  chose to pass instanceId.

Tool schemas updated to mark instanceId required; regression tests
added for both behaviors.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants