docs: add A27 runner-doctor catalog entry for ARC/DinD safe-output staging fix - #8969
Merged
Merged
Conversation
Copilot
AI
changed the title
[WIP] Update Runner Doctor knowledge-base with A27 ARC/DinD safe-output fix
docs: add A27 runner-doctor catalog entry for ARC/DinD safe-output staging fix
Sep 24, 2026
lpcox
marked this pull request as ready for review
September 24, 2026 19:35
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The runner-doctor alignment test lacks the customary A27 assertions.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Documents the A27 ARC/DinD safe-output staging failure mode fixed by #8933.
Changes:
- Adds A27 to the canonical failure catalog and quick lookup.
- Mirrors A27 in the portable runner-doctor agent.
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds canonical A27 diagnostics. |
.github/agents/self-hosted-runner-doctor.md |
Mirrors A27 for portable use. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | A24 | On `runner.topology: arc-dind`, `docker compose up -d --pull never` fails with `error mounting "/dev/null" to rootfs at ".../gh-aw/home/.npmrc": create mountpoint for .../.npmrc mount: ... openat .npmrc: read-only file system` — distinct from A23 in that it persists even after github/gh-aw-firewall#7998 (A23's fix), specifically when the credential mountpoint is missing under a declared-`rw` home bind whose real backing directory is genuinely read-only on the ARC/DinD-staged filesystem | `pruneUnmountableCredentialOverlays` (`src/services/agent-volumes/credential-hiding.ts`) decided whether a `/dev/null` credential mask could be mounted purely from the *declared* compose bind mode (`ro`/`rw`), never checking the real filesystem when the mode was `rw`. AWF's own home-directory mount is declared `rw`, but under `--docker-host-path-prefix` it can resolve to a directory that is genuinely read-only on the runner's staged filesystem; Docker doesn't remount that case read-only inside the container (unlike a declared `ro` bind), so it touches the real host path directly when creating a missing mountpoint and hits `EROFS`, crashing the agent container before it starts. | **Fixed in AWF (PR github/gh-aw-firewall#8086, merged 2026-09-04):** an overlay whose mountpoint already exists is always kept (mounting over an existing path succeeds regardless of declared mode or real writability); a missing mountpoint under a declared-`ro` bind is still always dropped; a missing mountpoint under a declared-`rw` bind is now probed against the real filesystem (walking up to the nearest existing ancestor, mirroring how Docker creates missing intermediate directories) and the overlay is dropped if that real directory isn't writable. Upgrade AWF to include github/gh-aw-firewall#8086. | Reproduce with an absent credential path (for example, `.npmrc`) beneath a `chmod`-based real read-only home directory under `runner.topology: arc-dind` + `--docker-host-path-prefix`; on unpatched AWF (even with github/gh-aw-firewall#7998 applied) `docker compose up` fails with the EROFS mounting error above; on patched AWF the missing overlay is skipped and the agent starts. Existing credential paths remain mountable. | github/gh-aw#57468, github/gh-aw-firewall#8076, github/gh-aw-firewall#8086 | | ||
| | A25 | On `runner.topology: arc-dind`, workloads inside the AWF sandbox need a GitHub Actions `services:` container's native protocol (DB drivers, migration tools, etc.) but cannot reach it — the `services:` container runs on the runner's own bridge network while the AWF agent runs on the isolated `awf-net`, and the two bridges are unrouted; existing host-iptables service-port routing doesn't help because ARC/DinD network isolation never programs host iptables rules | No AWF mechanism previously joined a `services:` container to `awf-net`; raw-protocol clients (e.g. `psql`) have no route from the sandbox to the service | **Documented in AWF (PR github/gh-aw-firewall#8085, merged 2026-09-04)**: new `docs/arc-dind.md` section "Joining `services:` containers to `awf-net` for direct protocol access" documents a verified workaround — a pre-step waits for `awf-net` to exist, then `docker network connect --alias <name> awf-net <service_container>` attaches the service container (never the agent) with a resolvable alias. Security invariant: only the service joins `awf-net`; joining the agent to the runner bridge would bypass the Squid egress firewall. Longer-term direction (`services.<name>.attach: true` compiler sugar) is not yet implemented. | Confirm the `services:` container, not the agent, is the one calling `docker network connect --alias <name> awf-net <container>`; verify the agent can resolve/reach `<name>` after the join; confirm the agent itself never appears attached to the runner's default bridge | github/gh-aw-firewall#8075, github/gh-aw-firewall#8085 | | ||
| | A26 | On `runner.topology: arc-dind` deployed on OpenShift/ARO clusters, Squid fails all `CONNECT` requests with `503 HIER_NONE`; the `awf-net` bridge subnet (`172.30.0.0/24`, Squid pinned at `172.30.0.10`) collides with the cluster's default service CIDR (`172.30.0.0/16`) and specifically with the CoreDNS ClusterIP (`172.30.0.10`), so Squid sends its own DNS queries to itself | `src/docker-manager.ts` hardcoded the `awf-net` subnet to `172.30.0.0/24` in `generateDockerCompose()` with no override, ignoring the Docker daemon's `--default-address-pool` because Compose-declared subnets take precedence; `src/squid-config.ts` templated `dns_nameservers` from the runner pod's `/etc/resolv.conf`, which resolves inside the claimed subnet on OpenShift/ARO, creating a self-referential route once `awf-net` exists | **Fixed in AWF (PR github/gh-aw-firewall#8398, merged 2026-09-10):** new `--network-subnet <cidr>` CLI flag / `network.subnet` config key relocates `awf-net` off the default `172.30.0.0/24` (accepts `/16`–`/26`; fixed host offsets — `.1` gateway, `.10` Squid, `.20` agent, `.30` api-proxy, `.40` DoH, `.50` cli-proxy — are rebased into the chosen block via `resolveNetworkAddressing()` in `src/network-subnet.ts`). `assertNetworkSubnetUsable()` now fails loudly at startup if the chosen (or default) subnet contains a detected/explicit DNS resolver or overlaps a non-Docker-managed host route, instead of silently producing broken DNS. The override is rejected with `--container-runtime sbx|cloud-hypervisor` because those runtimes have fixed guest network plans. Upgrade AWF to include github/gh-aw-firewall#8398. | Check `awf --version` for github/gh-aw-firewall#8398; on OpenShift/ARO inspect the startup error for `The awf-net subnet 172.30.0.0/24 contains the DNS resolver(s) ...`; relocate with `--network-subnet 10.88.0.0/24` (or `{"network":{"subnet":"10.88.0.0/24"}}`) and confirm Squid's `CONNECT` no longer returns `503 HIER_NONE` | github/gh-aw#59880, github/gh-aw-firewall#8390, github/gh-aw-firewall#8398 | | ||
| | A27 | On `runner.topology: arc-dind`, safe-output payload files staged under `/tmp/gh-aw/agent` (patch/bundle handoff, PR body, etc.) silently fail to write inside the AWF chroot — the heredoc/write succeeds on the runner side against a path the Docker daemon never sees, producing empty safe-output fields (e.g. an empty PR body) instead of a hard failure | ARC/DinD's "Create gh-aw temp directory" step and AWF's `dind-bootstrap.ts` pre-staging only created `/tmp/gh-aw/agent` on the runner filesystem. The Docker daemon in a split-filesystem DinD topology cannot see that path, so when `RUNNER_TEMP` is set (the daemon-visible staging root), the corresponding `${RUNNER_TEMP}/gh-aw/agent` directory was never created or pre-staged, leaving no daemon-visible landing path for the agent's safe-output writes | **Fixed in AWF (PR github/gh-aw-firewall#8933, merged 2026-09-24, fixes github/gh-aw-firewall#8932):** `ensureAgentStagingDirectories()` in `src/dind-bootstrap.ts` now creates both `/tmp/gh-aw/agent` (existing behavior) and `${RUNNER_TEMP}/gh-aw/agent` when `RUNNER_TEMP` is set; `agent` was added to `DEFAULT_PRE_STAGE_DIRS` so DinD bootstrap also pre-stages it through Docker; `${RUNNER_TEMP}/gh-aw` is pre-staged as a second work-directory tree when it differs from the configured `dind.workDir`. Non-DinD/no-`RUNNER_TEMP` behavior is unchanged. Upgrade AWF to include github/gh-aw-firewall#8933. | Set `RUNNER_TEMP` and `dind.preStageDirs: true` and inspect whether `${RUNNER_TEMP}/gh-aw/agent` exists and was pre-staged through Docker (not just `/tmp/gh-aw/agent`); on unpatched AWF only the runner-side path exists and safe-output payloads written to `/tmp/gh-aw/agent` inside the chroot silently vanish; on patched AWF both paths exist and are pre-staged | github/gh-aw#63045, github/gh-aw#62924, github/gh-aw-firewall#8932, github/gh-aw-firewall#8933, github/gh-aw-firewall#8938 | |
Collaborator
|
@copilot address the review feedback |
Contributor
Author
Contributor
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Contributor
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runner Doctor scanned recent AWF activity and identified one new ARC/DinD failure mode not yet in the self-hosted runner-doctor knowledge base: safe-output payloads staged under
/tmp/gh-aw/agentsilently vanish inside the AWF chroot whenRUNNER_TEMPis set, because the daemon-visible${RUNNER_TEMP}/gh-aw/agentpath was never created or pre-staged. The underlying code fix (ensureAgentStagingDirectories()insrc/dind-bootstrap.ts) already merged in #8933 — this PR is the documentation follow-up to record it in the catalog.Catalog updates
.github/workflows/shared/self-hosted-failure-modes.md(canonical source, imported at runtime by theself-hosted-runner-doctor.mdworkflow):Portable agent mirror
.github/agents/self-hosted-runner-doctor.md: mirrored both edits above exactly, keeping the embedded catalog copy in sync with the shared sourceNo source changes were needed — the fix itself (
DEFAULT_PRE_STAGE_DIRSincludingagent, dual staging of/tmp/gh-aw/agentand${RUNNER_TEMP}/gh-aw/agent) shipped in #8933; this PR only closes the documentation gap so future triage recognizes the signature (empty safe-output field, no write error, payload staged under/tmp/gh-aw/agent) as A27.