Sync D14 runner-doctor catalog with #8801 Cloud Hypervisor retry-fix follow-up - #8854
Merged
Merged
Conversation
Copilot
AI
changed the title
[WIP] Update Runner Doctor knowledge-base for D14 retry fix
Sync D14 runner-doctor catalog with #8801 Cloud Hypervisor retry-fix follow-up
Sep 21, 2026
lpcox
marked this pull request as ready for review
September 22, 2026 02:18
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The mirrored documentation says three retries, while the implementation performs three total attempts and therefore at most two retries.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 3
Open (3)
What changed in this PR
Synchronizes the runner-doctor D14 guidance with Cloud Hypervisor retry behavior from #8801.
Changes:
- Updates D14 guidance and citations across all three mirrors.
- Updates the portable-agent synchronization test.
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Updates D14 catalog and lookup. |
.github/workflows/self-hosted-runner-doctor.md |
Updates D14 playbook guidance. |
.github/agents/self-hosted-runner-doctor.md |
Synchronizes portable guidance. |
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Updates D14 citation assertion. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252, github/gh-aw-firewall#8568, github/gh-aw-firewall#8575 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks. A persistent result means the long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint; start/restart it with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway). **Additional fix (PR github/gh-aw-firewall#8575, merged 2026-09-15):** a transient Squid-startup race can produce one bypass hit while proxy chaining comes up, so `assertSbxEgressEnforced` retries up to 3 times, 2 seconds apart. Probe execution failures still throw immediately and persistent bypasses still fail closed. See `docs/sbx-integration.md` and inspect `sbx daemon status` for the daemon-level contract. | ||
|
|
||
| D14 / github/gh-aw-firewall#8620, github/gh-aw-firewall#8622 — On `--container-runtime cloud-hypervisor`, the secure-launcher version probe could abort with `"cloud-hypervisor --version" exited with code undefined` when a staged binary was signal-terminated, missing/not executable, or incomplete. **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** probe errors report both `exitCode` and `signalCode`; zero-size staged artifacts are rejected before hashing/execution; unsupported runner/KVM hosts raise `CloudHypervisorUnsupportedHostError` and fall back to Docker. Artifact-trust, digest, version, and configuration failures remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. | ||
| D14 / github/gh-aw-firewall#8620, github/gh-aw-firewall#8622, github/gh-aw-firewall#8727, github/gh-aw-firewall#8728, github/gh-aw-firewall#8767, github/gh-aw-firewall#8801 — On `--container-runtime cloud-hypervisor`, the secure-launcher version probe could abort with `"cloud-hypervisor --version" exited with code undefined` when a staged binary was signal-terminated, missing/not executable, or incomplete. **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** probe errors report both `exitCode` and `signalCode`; zero-size staged artifacts are rejected before hashing/execution; unsupported runner/KVM hosts raise `CloudHypervisorUnsupportedHostError` and fall back to Docker. Artifact-trust, digest, version, and configuration failures remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. **Further fixed in AWF (PR github/gh-aw-firewall#8801, merged 2026-09-20):** the `--version` probe for both the Cloud Hypervisor and virtiofsd binaries is now retried up to 3 times (250 ms apart) before failing, since the staged artifact's digest is already trusted by this point and a single failure is more likely a transient exec hiccup. Deterministic errors (`ENOENT`/`EACCES`/`EISDIR`/`ENOTDIR`/`ENOEXEC`) skip the retry and fail closed immediately. Upgrade AWF to include github/gh-aw-firewall#8801. |
| D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252, github/gh-aw-firewall#8568, github/gh-aw-firewall#8575 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks. A persistent result means the long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint; start/restart it with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway). **Additional fix (PR github/gh-aw-firewall#8575, merged 2026-09-15):** a transient Squid-startup race can produce one bypass hit while proxy chaining comes up, so `assertSbxEgressEnforced` retries up to 3 times, 2 seconds apart. Probe execution failures still throw immediately and persistent bypasses still fail closed. See `docs/sbx-integration.md` and inspect `sbx daemon status` for the daemon-level contract. | ||
|
|
||
| D14 / github/gh-aw-firewall#8620, github/gh-aw-firewall#8622, github/gh-aw-firewall#8727, github/gh-aw-firewall#8728 — On `--container-runtime cloud-hypervisor`, the secure-launcher version probe could abort with `"cloud-hypervisor --version" exited with code undefined` when a staged binary was signal-terminated, missing/not executable, or incomplete. **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** probe errors report both `exitCode` and `signalCode`; zero-size staged artifacts are rejected before hashing/execution; unsupported runner/KVM hosts raise `CloudHypervisorUnsupportedHostError` and fall back to Docker. Artifact-trust, digest, version, and configuration failures remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. | ||
| D14 / github/gh-aw-firewall#8620, github/gh-aw-firewall#8622, github/gh-aw-firewall#8727, github/gh-aw-firewall#8728, github/gh-aw-firewall#8767, github/gh-aw-firewall#8801 — On `--container-runtime cloud-hypervisor`, the secure-launcher version probe could abort with `"cloud-hypervisor --version" exited with code undefined` when a staged binary was signal-terminated, missing/not executable, or incomplete. **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** probe errors report both `exitCode` and `signalCode`; zero-size staged artifacts are rejected before hashing/execution; unsupported runner/KVM hosts raise `CloudHypervisorUnsupportedHostError` and fall back to Docker. Artifact-trust, digest, version, and configuration failures remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. **Further fixed in AWF (PR github/gh-aw-firewall#8801, merged 2026-09-20):** the `--version` probe for both the Cloud Hypervisor and virtiofsd binaries is now retried up to 3 times (250 ms apart) before failing, since the staged artifact's digest is already trusted by this point and a single failure is more likely a transient exec hiccup. Deterministic errors (`ENOENT`/`EACCES`/`EISDIR`/`ENOTDIR`/`ENOEXEC`) skip the retry and fail closed immediately. Upgrade AWF to include github/gh-aw-firewall#8801. |
| | D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 | | ||
| | D13 | Under `--container-runtime sbx` with `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF reports `Direct sbx egress reached 1.1.1.1 without proxy environment variables` and aborts before the agent starts, even though Squid is healthy (`squid host.docker.internal:3128 -> 200`). | The long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint, allowing its sandboxed egress to bypass Squid. AWF correctly detects this and fails closed. A transient Squid-startup race can produce the same probe result before the sbx daemon's proxy chaining is ready. | The caller/workflow owns daemon lifecycle: start or restart `sbx daemon` with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway) before AWF creates the sandbox. PR github/gh-aw-firewall#8252 fixes this repository's smoke-workflow post-processing final daemon restart. **Additional fix (PR github/gh-aw-firewall#8575, merged 2026-09-15):** `assertSbxEgressEnforced` retries a bypass hit up to 3 times, 2 seconds apart, to absorb the transient Squid-startup race; probe execution failures still throw immediately and a persistent bypass still fails closed. See `docs/sbx-integration.md` for the general contract. | Run `sbx daemon status` and inspect its environment for `DOCKER_SANDBOXES_PROXY`; restart it without that variable and confirm AWF's direct-egress probe reaches `1.1.1.1` or a denied destination without proxy variables. If the result clears after a brief retry while Squid starts, upgrade for github/gh-aw-firewall#8575. | github/gh-aw-firewall#8250, github/gh-aw-firewall#8252, github/gh-aw-firewall#8568, github/gh-aw-firewall#8575 | | ||
| | D14 | On `--container-runtime cloud-hypervisor`, the agent run aborts before the engine starts with `"/run/awf-cloud-hypervisor/trusted-artifacts/run-<id>/cloud-hypervisor --version" exited with code undefined`; the failure propagates and kills the whole engine run even on hosts where Cloud Hypervisor is simply unsupported (no KVM) or the staged artifact is incomplete | The Cloud Hypervisor secure-launcher version probe only captured `exitCode`; when the child was signal-terminated, never spawned, or ran against a corrupted/partial staged artifact, `exitCode` came back `null`/`undefined`, leaving no way to distinguish those conditions from a genuine version-check failure | **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** the version probe reports both `exitCode` and `signalCode`; staged trusted artifacts must be nonzero size before SHA-256 hashing/execution; `CloudHypervisorUnsupportedHostError` falls back to Docker on unsupported runner/KVM hosts. Artifact-trust, digest, version, and configuration errors remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. | Reproduce on a host without KVM/nested virtualization using `--container-runtime cloud-hypervisor`; on unpatched AWF the probe reports `exited with code undefined`; on patched AWF confirm signal termination names `signalCode`, or an unsupported host warns and falls back to Docker. | github/gh-aw-firewall#8620, github/gh-aw-firewall#8622, github/gh-aw-firewall#8727, github/gh-aw-firewall#8728 | | ||
| | D14 | On `--container-runtime cloud-hypervisor`, the agent run aborts before the engine starts with `"/run/awf-cloud-hypervisor/trusted-artifacts/run-<id>/cloud-hypervisor --version" exited with code undefined`; the failure propagates and kills the whole engine run even on hosts where Cloud Hypervisor is simply unsupported (no KVM) or the staged artifact is incomplete | The Cloud Hypervisor secure-launcher version probe only captured `exitCode`; when the child was signal-terminated, never spawned, or ran against a corrupted/partial staged artifact, `exitCode` came back `null`/`undefined`, leaving no way to distinguish those conditions from a genuine version-check failure | **Fixed in AWF (PR github/gh-aw-firewall#8622, merged 2026-09-16):** the version probe reports both `exitCode` and `signalCode`; staged trusted artifacts must be nonzero size before SHA-256 hashing/execution; `CloudHypervisorUnsupportedHostError` falls back to Docker on unsupported runner/KVM hosts. Artifact-trust, digest, version, and configuration errors remain fail-closed. Upgrade AWF to include github/gh-aw-firewall#8622. **Further fixed in AWF (PR github/gh-aw-firewall#8801, merged 2026-09-20):** the `--version` probe for both the Cloud Hypervisor and virtiofsd binaries is now retried up to 3 times (250 ms apart) before failing, since the staged artifact's digest is already trusted by this point and a single failure is more likely a transient exec hiccup. Deterministic errors (`ENOENT`/`EACCES`/`EISDIR`/`ENOTDIR`/`ENOEXEC`) skip the retry and fail closed immediately. Upgrade AWF to include github/gh-aw-firewall#8801. | Reproduce on a host without KVM/nested virtualization using `--container-runtime cloud-hypervisor`; on unpatched AWF the probe reports `exited with code undefined`; on patched AWF confirm signal termination names `signalCode`, or an unsupported host warns and falls back to Docker; on AWF including github/gh-aw-firewall#8801, a single transient probe failure no longer aborts the run — check agent/preflight logs for `Cloud Hypervisor version probe failed for "<path>" (attempt N/3)` warnings indicating the retry engaged, versus an immediate failure for the fail-fast error codes above | github/gh-aw-firewall#8620, github/gh-aw-firewall#8622, github/gh-aw-firewall#8727, github/gh-aw-firewall#8728, github/gh-aw-firewall#8767, github/gh-aw-firewall#8801 | |
Collaborator
|
@copilot Fix the code for all comments in this review thread. When a review comment includes a suggested change, apply the suggestion exactly. Do not make changes beyond what is described in the linked review thread. |
Contributor
Author
I’m blocked from completing this because the linked review requires changes in |
Contributor
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Contributor
|
⏳ Copilot review left inline comments. @copilot To proceed:
|
Contributor
|
| Metric | Base | PR | Delta |
|---|---|---|---|
| Lines | 93.28% | 93.11% | 📉 -0.17% |
| Statements | 91.94% | 91.69% | 📉 -0.25% |
| Functions | 92.00% | 91.79% | 📉 -0.21% |
| Branches | 85.64% | 85.26% | 📉 -0.38% |
📁 Per-file Coverage Changes (4 files)
| File | Lines (Before → After) | Statements (Before → After) |
|---|---|---|
src/nvx/filesystem-builder.ts |
88.4% → 88.5% (+0.05%) | 87.5% → 87.6% (+0.05%) |
src/nvx/one-shot-adapter.ts |
80.1% → 80.4% (+0.30%) | 77.4% → 77.7% (+0.29%) |
src/nvx/cleanup-record.ts |
83.3% → 85.1% (+1.77%) | 82.5% → 84.4% (+1.82%) |
src/log-directory-setup.ts |
96.2% → 100.0% (+3.78%) | 96.3% → 100.0% (+3.71%) |
✨ New Files (1 files)
src/nvx/runtime-lifecycle.ts: 84.5% lines
Coverage comparison generated by scripts/ci/compare-coverage.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The self-hosted runner-doctor knowledge base's D14 entry was missing coverage of PR #8801, which added a bounded retry for transient Cloud Hypervisor/virtiofsd
--versionprobe failures during preflight, leaving the shared catalog, workflow playbook, and portable agent copy out of sync.Catalog updates
.github/workflows/shared/self-hosted-failure-modes.md: extended the D14 row's Fix/flag, Probe, and Citations columns, plus the error-string quick-lookup entry, with the#8801retry-fix details..github/workflows/self-hosted-runner-doctor.md: mirrored the same D14 playbook paragraph and error-string lookup update..github/agents/self-hosted-runner-doctor.md: applied identical edits across all 4 embedded mirror locations (top lookup, playbook paragraph, embedded table row, embedded lookup row), and aligned the citation lists (#8620, #8622, #8727, #8728, #8767, #8801) so all three copies match exactly.New lesson captured
ENOENT/EACCES/EISDIR/ENOTDIR/ENOEXEC) still fail fast; only transient exec hiccups are retried (3 attempts, 250ms apart), since the artifact digest is already trusted by the time the probe runs.Cloud Hypervisor version probe failed for "<path>" (attempt N/3)warning signature for diagnosing engaged retries vs. immediate fail-fast errors.Test alignment
scripts/ci/self-hosted-runner-doctor-workflow.test.ts: updated the regex assertion checking the D14 row's citation-column ending, which previously expected#8622and now expects#8801.