feat: add NVX build-test smoke workflow by lpcox · Pull Request #9014 · github/gh-aw-firewall

lpcox · 2026-09-25T21:11:00Z

Summary

Adds smoke-nvx-build-test, an NVX counterpart to the existing smoke-cloud-hypervisor-build-test and smoke-gvisor-build-test workflows. It validates that the NVX microVM runtime can run real Node.js and Go build/test workloads on KVM hardware — a much heavier workload than the single Copilot prompt that smoke-nvx-copilot exercises.

Design

Workload parity with Cloud Hypervisor. The guest runs the same workload, with the same result schema and post-step validation:

GitHub.com connectivity (http_code must be 200)
npm ci && npm run build
A Jest subset (squid-config|docker-manager|logger)
Go build and test of the pinned color and uuid fixtures
Network isolation (example.com must be blocked)

NVX-specific structure. NVX is not a native gh-aw sandbox.agent.runtime, so this workflow follows smoke-nvx-copilot instead of setting a runtime in the frontmatter:

A build_nvx_artifacts job fetches and verifies the pinned NVX release, then self-attests the manifest.
A pre-agent step runs the workload through an inner awf --container-runtime nvx.
The agent only analyzes the recorded evidence.

Guest layer. Alpine 3.22.1 (the same pinned digest as the Copilot smoke) with Node.js 22 and Go 1.24:

Toolchains are installed on the host before the microVM exists, so the guest never needs package-manager egress.
The guest egress allowlist is only github.com,registry.npmjs.org. The Go fixtures have no module dependencies, and GOTOOLCHAIN=local blocks toolchain downloads.
I checked the lockfile's native dependencies (esbuild, unrs-resolver, @parcel/watcher); all ship musl builds.

Host checkout safety. The guest builds from a scratch copy of the workspace (excluding node_modules, .git, and dist). The host checkout and its glibc node_modules are never modified. Only a .nvx-build-test/ results directory comes back through the workspace export.

Sizing. NVX's defaults (512 MiB, 128 pids) are sized for one CLI invocation, and the guest is hard-coded to 1 vCPU. This workflow requests:

4 GiB guest memory and a 3.5 GiB cgroup ceiling
1024 pids
A 6 GiB scratch overlay (the ceiling is 8 GiB)

Jest runs --runInBand to fit the single vCPU.

Extra checks beyond the Cloud Hypervisor workflow. microvm_run (awf exit code), guest_completed, and workspace_copy_back. The last one proves both --env passthrough and the copy-back path: the guest writes a per-run marker and the host checks it.

Evidence. All logs (awf.log, inner proxy logs, and per-stage workload logs) are uploaded as the nvx-build-test-evidence artifact.

Source change

src/nvx/artifact-manifest.ts adds NVX_BUILD_TEST_SIGNER_WORKFLOW, so the new workflow can verify its own self-attested manifest. This mirrors the existing NVX_SMOKE_SIGNER_WORKFLOW entry: it trusts exactly one additional pinned lock file and leaves the default signer unchanged. The preflight test is parameterized to cover both smoke signers. The existing untrusted-signer rejection test still passes.

Validation

gh aw compile smoke-nvx-build-test: 0 warnings. Postprocessing touched only the new lock file, and the session-state fix from fix: get the NVX Copilot smoke test to genuinely pass on real KVM hardware #9010 applies to it automatically.
The guest, layer-build, and run scripts were extracted from the compiled lock file and pass bash -n. Both heredocs and the RESULTS_EOF terminator land at column 0.
jest src/nvx/preflight.test.ts: 24 of 24 tests pass. tsc --noEmit is clean. ESLint shows no new warnings.

Not yet run on hardware. I couldn't run it locally. The end-to-end check is running the workflow on this PR. Because the workflow isn't on main yet, workflow_dispatch may not be able to target it; adding the test-nvx-build label to this PR triggers it instead.

Known risk

The NVX confinement verifier race (#9012) affects every NVX launch, so this workflow can fail at openvmmReady before the guest boots, for reasons unrelated to this change. In that case the microvm_run and guest_completed checks fail, with the confinement error visible in logs/awf.log.

Add smoke-nvx-build-test, modeled on smoke-cloud-hypervisor-build-test, to validate that the NVX microVM runtime can run real Node.js and Go build/test workloads on KVM hardware. The workload runs inside an NVX one-shot microVM from a pre-agent step (NVX is not a native gh-aw sandbox runtime), using an Alpine 3.22.1 guest layer with Node.js 22 and Go 1.24 preinstalled on the host. It builds from a scratch copy of the workspace so the host checkout is never modified, and returns results through the workspace copy-back. Trust the new workflow as a pinned NVX artifact signer, mirroring the existing smoke-nvx-copilot entry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-25T21:11:38Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-25T21:11:48Z

🧊🏗️ Smoke NVX Build Test reports failed. NVX compatibility issue detected.

🧊🏗️ NVX build test by Smoke NVX Build Test

github-actions · 2026-09-25T21:15:04Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	92.65%	92.65%	➡️ +0.00%
Statements	91.13%	91.13%	➡️ +0.00%
Functions	89.11%	89.11%	➡️ +0.00%
Branches	84.26%	84.25%	📉 -0.01%

📁 Per-file Coverage Changes (3 files)

File	Lines (Before → After)	Statements (Before → After)
`src/nvx/one-shot-adapter.ts`	83.5% → 82.9% (-0.60%)	80.3% → 79.8% (-0.56%)
`src/nvx/artifact-manifest.ts`	87.9% → 88.0% (+0.18%)	87.9% → 88.0% (+0.18%)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Copilot review overview

🟡 Changes recommended

The hardware run exhausts runner disk before boot, and guest npm egress excludes the lockfile’s package registry.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2

Open (2)

npm ci cannot access lockfile package URLs under the NVX allowlist · New
6 GiB scratch filesystem exceeds available runner disk space · New

What changed in this PR

Adds an NVX smoke workflow for real Node.js and Go build/test workloads inside a microVM.

Changes:

Adds and compiles the NVX build-test workflow.
Trusts its pinned workflow as an NVX artifact signer.
Extends preflight signer tests.

File	Description
`.github/workflows/smoke-nvx-build-test.md`	Defines the smoke workload and validation.
`.github/workflows/smoke-nvx-build-test.lock.yml`	Compiled workflow.
`src/nvx/artifact-manifest.ts`	Adds the trusted signer identity.
`src/nvx/preflight.test.ts`	Tests both smoke signers.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        -cf - . | tar -C "$build" -xf -
+      cd "$build"
+
+      timeout 15m npm ci > "$out/npm-ci.log" 2>&1


+        --nvx-memory-mib 4096 \
+        --nvx-memory-max-bytes 3758096384 \
+        --nvx-pids-max 1024 \
+        --nvx-scratch-bytes 6442450944 \


NVX stages its EROFS layer images and the guest scratch image under /run/awf-nvx/runs, and /run is a RAM-backed tmpfs Ubuntu sizes at ~10% of RAM. The first hardware run failed in mke2fs with 'No space left on device' before the guest booted. Grow the tmpfs ceiling (pages are only allocated when written) and shrink the scratch overlay to 3 GiB. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-25T21:19:57Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-09-25T21:20:12Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-25T21:20:26Z

🧊🏗️ Smoke NVX Build Test reports failed. NVX compatibility issue detected.

🧊🏗️ NVX build test by Smoke NVX Build Test

lpcox · 2026-09-25T21:44:13Z

@copilot address review feedback

Copilot · 2026-09-25T21:52:03Z

@copilot address review feedback

Both findings are now addressed:

npm registry (fixed in 2856d0a): In the guest's scratch copy, the lockfile's Azure-mirror resolved URLs are now rewritten to registry.npmjs.org, and the .npmrc registry is overridden. The host checkout isn't touched. The reviewer's suggestion, replace-registry-host=always, doesn't work here: it swaps only the host and keeps the mirror's /1es-public/.../npm/registry/ path, so every tarball 404s. I checked it locally. After the rewrite, npm ci against the public registry installs all 863 packages with matching integrity, and the rewrite also works under Alpine's busybox sed.
Disk space: 599f28b already fixed this by resizing the /run tmpfs and shrinking scratch to 3 GiB. Run 36190954018 got past mke2fs. It then failed at openvmmReady on the known confinement race (NVX/Cloud Hypervisor confinement verifier fails intermittently on a thread-set TOCTOU race #9012), before the guest booted, so the workload hasn't run on hardware yet.

github-actions · 2026-09-25T21:55:49Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-25T21:55:58Z

🧊🏗️ Smoke NVX Build Test reports failed. NVX compatibility issue detected.

🧊🏗️ NVX build test by Smoke NVX Build Test

github-actions · 2026-09-25T22:06:49Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	92.65%	92.65%	➡️ +0.00%
Statements	91.13%	91.13%	➡️ +0.00%
Functions	89.11%	89.11%	➡️ +0.00%
Branches	84.26%	84.25%	📉 -0.01%

📁 Per-file Coverage Changes (3 files)

File	Lines (Before → After)	Statements (Before → After)
`src/nvx/one-shot-adapter.ts`	83.5% → 82.9% (-0.60%)	80.3% → 79.8% (-0.56%)
`src/nvx/artifact-manifest.ts`	87.9% → 88.0% (+0.18%)	87.9% → 88.0% (+0.18%)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

NVX applies the credential deny list to every layer, including the workspace, so the repo's .npmrc (legacy-peer-deps=true) never reaches the guest and npm ci fails with ERESOLVE on the @babel/core 7/8 peer conflict. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-25T22:07:33Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-25T22:07:45Z

🧊🏗️ Smoke NVX Build Test reports failed. NVX compatibility issue detected.

🧊🏗️ NVX build test by Smoke NVX Build Test

github-actions · 2026-09-25T22:11:03Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	92.65%	92.65%	➡️ +0.00%
Statements	91.13%	91.13%	➡️ +0.00%
Functions	89.11%	89.17%	📈 +0.06%
Branches	84.26%	84.22%	📉 -0.04%

📁 Per-file Coverage Changes (5 files)

File	Lines (Before → After)	Statements (Before → After)
`src/nvx/one-shot-adapter.ts`	83.5% → 82.9% (-0.60%)	80.3% → 79.8% (-0.56%)
`src/nvx/artifact-manifest.ts`	87.9% → 88.0% (+0.18%)	87.9% → 88.0% (+0.18%)
`src/nvx/confinement.ts`	88.2% → 88.9% (+0.73%)	83.3% → 84.6% (+1.24%)
`src/cloud-hypervisor/confinement-verifier.ts`	82.9% → 84.7% (+1.83%)	81.6% → 83.0% (+1.39%)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Jest ran 395/400 in the guest. The remaining failures came from the guest layer: /etc/passwd had no root entry (getRealUserHome tests) and /var/tmp was not world-writable (enclave path tests). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-25T22:15:47Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-09-25T22:16:03Z

🧊🏗️ Smoke NVX Build Test completed. NVX build test passed. ✅

🧊🏗️ NVX build test by Smoke NVX Build Test

github-actions · 2026-09-25T22:19:16Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	92.65%	92.65%	➡️ +0.00%
Statements	91.13%	91.13%	➡️ +0.00%
Functions	89.11%	89.17%	📈 +0.06%
Branches	84.26%	84.22%	📉 -0.04%

📁 Per-file Coverage Changes (5 files)

File	Lines (Before → After)	Statements (Before → After)
`src/nvx/one-shot-adapter.ts`	83.5% → 82.9% (-0.60%)	80.3% → 79.8% (-0.56%)
`src/nvx/artifact-manifest.ts`	87.9% → 88.0% (+0.18%)	87.9% → 88.0% (+0.18%)
`src/nvx/confinement.ts`	88.2% → 88.9% (+0.73%)	83.3% → 84.6% (+1.24%)
`src/cloud-hypervisor/confinement-verifier.ts`	82.9% → 84.7% (+1.83%)	81.6% → 83.0% (+1.39%)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-09-25T22:23:12Z

🧊🏗️ NVX Build Test Results

Test	Status
microVM run	✅
Workspace copy-back	✅
GitHub.com connectivity	✅
Node.js build (`npm ci && npm run build`)	✅
Node.js tests (Jest subset)	✅
Go build (color, uuid)	✅
Go tests (color, uuid)	✅
Network isolation	✅

Overall: PASS

🧊🏗️ NVX build test by Smoke NVX Build Test
_{Add label test-nvx-build to run again}

lpcox · 2026-09-25T22:23:49Z

✅ Validated on real KVM hardware: run 36195809798 passed every check (microvm_run, guest_completed, workspace_copy_back, node_build, node_test, go_build, go_test, network_isolation).

Fixes needed to get here:

fix: tolerate benign VMM thread churn in confinement re-verification #9017 (merged): the confinement verifier failed on normal OpenVMM thread churn.
115a3a74: restate legacy-peer-deps in the guest. NVX strips credential-named files such as .npmrc from every layer, including the workspace, so the repo .npmrc never reaches the guest and npm ci fails with ERESOLVE.
993ce865: give the guest a standard /etc/passwd (with a root entry) and a world-writable /var/tmp. Jest had gone from 395/400 to 400/400.

Copilot AI balanced review requested due to automatic review settings September 25, 2026 21:11

lpcox added the test-nvx-build Run the NVX build-test smoke workflow label Sep 25, 2026

Copilot started reviewing on behalf of lpcox September 25, 2026 21:11 View session

This comment has been minimized.

Sign in to view

github-actions Bot mentioned this pull request Sep 25, 2026

[aw] Smoke NVX Build Test failed #9015

Closed

Copilot AI reviewed Sep 25, 2026

View reviewed changes

lpcox added test-nvx-build Run the NVX build-test smoke workflow and removed test-nvx-build Run the NVX build-test smoke workflow labels Sep 25, 2026

This comment has been minimized.

Sign in to view

lpcox mentioned this pull request Sep 25, 2026

fix: tolerate benign VMM thread churn in confinement re-verification #9017

Merged

Copilot started work on behalf of lpcox September 25, 2026 21:44 View session

fix: rewrite lockfile mirror URLs for NVX guest npm ci

2856d0a

Copilot finished work on behalf of lpcox September 25, 2026 21:52

lpcox added test-nvx-build Run the NVX build-test smoke workflow and removed test-nvx-build Run the NVX build-test smoke workflow labels Sep 25, 2026

This comment has been minimized.

Sign in to view

lpcox added test-nvx-build Run the NVX build-test smoke workflow and removed test-nvx-build Run the NVX build-test smoke workflow labels Sep 25, 2026

This comment has been minimized.

Sign in to view

lpcox added test-nvx-build Run the NVX build-test smoke workflow and removed test-nvx-build Run the NVX build-test smoke workflow labels Sep 25, 2026

github-actions Bot added the smoke-nvx-build label Sep 25, 2026

lpcox merged commit 7d99828 into main Sep 25, 2026
168 of 169 checks passed

lpcox deleted the add-nvx-build-test-smoke branch September 25, 2026 22:36

github-actions Bot mentioned this pull request Sep 26, 2026

Prevent thread churn from aborting microVM confinement verification #9016

Merged

Conversation

lpcox commented Sep 25, 2026

Summary

Design

Source change

Validation

Known risk

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

lpcox commented Sep 25, 2026

Uh oh!

Copilot AI commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 25, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Sep 25, 2026

🧊🏗️ NVX Build Test Results

Uh oh!

lpcox commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Sep 25, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading

github-actions Bot commented Sep 25, 2026 •

edited

Loading