From 0289f160d1b98ef267763857eef1698c37bdb99e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 17:12:51 +0000 Subject: [PATCH 1/6] Draft a plan for a 9p backend (vagrant/QEMU-style virtio-9p + diod tcp) Design notes for adding 9p as a matrix row: run the suite inside a virtme-ng guest against a QEMU -virtfs export (the actual vagrant-libvirt synced-folder stack), with a diod-over-TCP localhost variant mirroring the NFS backend where host kernels allow it. Covers backend-script knobs, matrix/CI wiring (per-row runs-on, pinned guest kernel), provisioning, a Phase-0 feasibility probe for the open runner questions (/dev/kvm, 9p modules on azure kernels), and PR sequencing. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- drafts/9p-backend-plan.md | 297 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 drafts/9p-backend-plan.md diff --git a/drafts/9p-backend-plan.md b/drafts/9p-backend-plan.md new file mode 100644 index 0000000..5ab819b --- /dev/null +++ b/drafts/9p-backend-plan.md @@ -0,0 +1,297 @@ +# Plan: a 9p backend (`bin/eval-under-9p`) + +Status: **plan, not yet implemented**. This documents the design and the +sequencing for adding 9p as a matrix row, so the implementation PRs can +be reviewed against something. Move the durable parts (settings tables, +known-red reasoning) into GOTCHAS.md as they land; delete this file when +the last phase lands or is explicitly dropped. + +## Why 9p + +9p is the filesystem people get, usually without choosing it, whenever a +directory is shared *into* a VM or container boundary: + +- **Vagrant + QEMU/libvirt**: `synced_folder ..., type: "9p"` in + vagrant-libvirt is QEMU's virtio-9p device (`-virtfs local,...`) on + the host side and the kernel's `v9fs` client (`mount -t 9p -o + trans=virtio`) on the guest side. This is the concrete case that + motivates the backend: our own Vagrantfile deliberately avoids 9p for + syncing ("rsync is more portable than 9p/virtiofs and avoids + permission surprises") -- those permission surprises are precisely + what this harness exists to measure rather than avoid. +- **WSL2**: `/mnt/c` and friends are 9p (Microsoft's own server). +- **Chrome OS crostini**, **kata-containers** (pre-virtiofs), various + lightweight-VM dev environments. + +The quirk classes are distinct from anything the current rows cover: +cache-mode staleness (`cache=loose` shows other-writer changes late; the +default no-cache mode makes some mmap patterns impossible), byte-range / +BSD locking that historically returns `ENOLCK` or is faked +server-side, no inotify propagation, ownership semantics that depend on +the *server's* security model rather than the client mount, `msize` +throughput cliffs, no `O_TMPFILE`. git-annex leans on locking, mmap +(via git), and HOME-relative sockets -- a 9p row should light up in +informative ways, per layer, exactly like the vfat row does. + +## The shape of the problem + +Every existing backend mounts on the host and runs the suite on the +host. The *interesting* 9p deployment splits across a VM boundary: the +server is the QEMU process on the host, the client is `v9fs` in the +guest kernel, and the transport is virtio. Testing "9p as vagrant users +experience it" therefore means running the suite **inside a guest**. +There are two mechanisms worth having, and they share one backend +script: + +### Transport `virtio` (primary -- the real vagrant/QEMU stack) + +Naively this inverts the harness: targets are installed on the runner +(`install-target.sh` builds into `$EVAL_UNDER_SRC_DIR`), but the suite +would have to run in a guest that has none of that. **virtme-ng** +dissolves the inversion: `vng` boots a QEMU guest whose root filesystem +is a copy-on-write view of the *host's own* root (virtiofs by default, +`--force-9p` as fallback), so runner-installed targets, the git-annex +daily build, `~/.gitconfig`, and the uid/gid layout all exist in the +guest unchanged. The backend then adds its own device for the +filesystem under test: + +- host side: a fresh scratch dir exported via + `-virtfs local,path=$ORIG,mount_tag=eval9p,security_model=,id=eval9p` + (passed through `vng --qemu-opts`); +- guest side: `mount -t 9p -o trans=virtio,version=9p2000.L[,msize=..][,cache=..] + eval9p $MNT`, then run the wrapped command with `TMPDIR` / + `DATALAD_TESTS_TEMP_DIR` / (`HOME` with `--set-home`) pointing at it. + +stdout/stderr stream to the job log as usual and `vng` propagates the +wrapped command's exit status (verify this early -- it is +load-bearing for CI redness). + +So the backend contract ("run CMD with TMPDIR on the mount") survives +intact; the only novelty is that CMD executes under a different kernel +instance. Writes to host paths *outside* the shared dirs land in the +CoW layer and evaporate -- which is a feature (free cleanup), except +for the git target's `t/test-results/**` that the workflow uploads: +pass `--rwdir "$EVAL_UNDER_SRC_DIR"` so those writes reach the host. + +### Transport `tcp` (secondary -- no VM, mirrors `eval-under-nfs`) + +`diod` (LLNL's 9P2000.L server; Ubuntu universe: 1.0.24-5 on jammy, +1.0.24-5.1 on noble) exports a fresh scratch dir on `127.0.0.1`, and +the host kernel mounts it: + + diod --foreground --no-auth --listen 127.0.0.1:5640 --export "$ORIG" & + mount -t 9p -o trans=tcp,port=5640,aname=$ORIG,version=9p2000.L,uname=root,access=user \ + 127.0.0.1 "$MNT" + +Same architecture as the NFS backend (localhost server + kernel client), +so it drops into the harness with zero conceptual novelty. It exercises +the same `v9fs` client code but a *different server* than QEMU's virtfs +-- different bug surface, cheaper row. It requires the 9p client +modules in the **host** kernel (see risks: azure kernels). + +**Recommendation:** implement `virtio` as the deliverable -- it is the +stack the backend exists to represent -- and `tcp` opportunistically; +the script skeleton (scratch dir, teardown trap, env plumbing, option +parsing) is shared, only `start_*`/`mount_*` differ per transport. + +## Backend script: `bin/eval-under-9p` + +House pattern (`set -eu`, `SUDO` arrays, `trap teardown EXIT`, here-doc +`usage()`, common flags on top). Backend-specific knobs, following the +"distro/kernel defaults on purpose, knobs to deviate" philosophy the +loop backend established: + +| Flag | Env var | Default | Purpose | +| --- | --- | --- | --- | +| `--transport {virtio,tcp}` | `EVAL_UNDER_9P_TRANSPORT` | `virtio` | Which of the two stacks above. | +| `--security-model M` | `EVAL_UNDER_9P_SECURITY_MODEL` | `mapped-xattr` | virtio only. QEMU virtfs server model: `mapped-xattr` (ownership/mode faked in xattrs; what vagrant-libvirt `accessmode: "mapped"` gives), `passthrough` (real uids; QEMU must run as root), `none`. | +| `--cache MODE` | `EVAL_UNDER_9P_CACHE` | unset (kernel default) | v9fs client cache mode. Unset = whatever the guest kernel defaults to; a knob because `loose` vs default is the single biggest semantic axis users hit. | +| `--msize BYTES` | `EVAL_UNDER_9P_MSIZE` | unset (kernel default; 128 KiB since 5.15) | Client request size; a throughput knob, occasionally a correctness one. | +| `--kernel VER` | `EVAL_UNDER_9P_KERNEL` | pinned in `.github/matrix.yaml` | virtio only. Guest kernel for `vng --run`. See "pin the guest kernel" below. | +| `--memory MB` / `--cpus N` | `EVAL_UNDER_9P_{MEMORY,CPUS}` | 8192 / nproc | virtio only. Guest sizing; public-repo runners have 4 vCPU / 16 GB. | +| `--port P` | `EVAL_UNDER_9P_PORT` | 5640 | tcp only. Non-privileged, non-564 so nothing collides. | + +Mount `version=9p2000.L` is fixed, not a knob: `.u` is legacy and diod +speaks only `.L`. + +Semantics to preserve from the existing backends: + +- **User identity.** Under `mapped-xattr` the server fabricates + ownership, so run the wrapped command as the invoking user (NFS-style + drop) -- that is what a vagrant user's synced folder looks like. + `needs-root` targets (pjdfstest, stress-ng) run as root *in the + guest*; under `mapped-xattr` their chown/mknod get absorbed into + xattr mapping -- measuring that is the point, and GOTCHAS must say so + before anyone reads those cells as kernel bugs. Under `passthrough`, + QEMU itself runs as root (we already have `sudo -E` in the workflow). +- **`--keep`.** tcp: leave diod + mount up, as NFS does. virtio: the + guest is gone when vng exits; keep the *host-side backing dir* + (readable directly; under `mapped-xattr` the real metadata sits in + `user.virtfs.*` xattrs) and echo the full `vng` command line so the + session can be relaunched interactively for poking. +- **Failure diagnostics.** The v9fs client logs to the *guest* dmesg, + which dies with the VM. Wrap the guest-side command so that on + non-zero exit it appends `dmesg | tail -50` to a host-visible file + (under the `--rwdir` or the 9p mount's backing dir), and teach + `bin/ci/dump-failure-logs.sh` to print it. Same lesson as + `wait_for_mount_usable()`: make the next red cell diagnosable from + the job log alone. +- **Mount-usability probe.** Reuse the BeeGFS create+write+read-back + probe inside the guest right after the mount, before handing over to + the suite. 9p mounts fail late and weird; a probe converts that into + an early loud error. + +Guest-side execution sketch (virtio), all inside one `vng` invocation +so there is exactly one boot per cell: + + vng --run "$KERNEL" --cpus "$CPUS" --memory "$MEM" \ + --rwdir "$EVAL_UNDER_SRC_DIR" \ + --qemu-opts "-virtfs local,path=$ORIG,mount_tag=eval9p,security_model=$SECMODEL,id=eval9p" \ + -- bin/eval-under-9p --guest-stage2 ... + +with `--guest-stage2` (hidden flag) doing: modprobe 9p/9pnet_virtio if +modular, mount, probe, mkdir RUN_HOME, exec the command with the env +trio, capture dmesg on failure. Re-entering the same script keeps the +host/guest halves in one reviewable file. + +## Matrix integration + +`.github/matrix.yaml` rows -- the `version` slot becomes the variant +token (slug-safe: no slashes, per the `cell_slug()` lesson): + + - backend: 9p + version: virtio-mapped + label: "9p virtio (mapped)" + - backend: 9p + version: virtio-passthrough + label: "9p virtio (passthrough)" + +Start by landing `virtio-mapped` only (4 new cells); add +`virtio-passthrough` once the first row's failure modes are understood, +and `tcp-diod` if/when the host-module probe says the runners can do it. +`run-under.sh` grows a `9p)` case that splits the version token into +`--transport` / `--security-model` flags, exactly parallel to the +`loop)` case translating `version` into `--fs`. + +**Pin the guest kernel.** v9fs client behavior moves significantly +between kernel versions (the 6.6-6.8 cache rework renamed and +re-defaulted the cache modes). An unpinned guest kernel makes a +newly-red cell ambiguous in exactly the way the pinned `refs:` exist to +prevent -- so add e.g. `refs: { 9p-kernel: "6.8" }` and have the +backend default `--kernel` from it (`vng --run ` fetches a +prebuilt kernel; the host's running kernel remains an explicit opt-in +via `--kernel host`). Bump deliberately, and expect GOTCHAS entries to +be keyed to it. + +**Per-row `runs-on`.** The `test` job is currently hard-coded to +ubuntu-22.04 for BeeGFS-DKMS reasons that do not bind the 9p rows, and +ubuntu-24.04 is a materially better host here: `virtme-ng` (1.22-1) and +rust `virtiofsd` are packaged, QEMU is 8.2, and the stock kernel is +newer. Add an optional `runs-on:` key per backend row (default +ubuntu-22.04), emit it from `matrix-json.sh`, and set +`runs-on: ${{ matrix.runs-on }}` in the workflow. Contained change, +keeps the BeeGFS rows untouched, and removes the need to pip-install +virtme-ng on jammy. + +`bin/ci/install-backend.sh` gains `install_9p()`: + +- `apt_install qemu-system-x86 qemu-utils virtme-ng` (24.04; on 22.04 + fall back to `pipx install virtme-ng`), plus `diod` when the tcp + variant lands; +- best-effort `apt_install linux-modules-extra-$(uname -r)` -- + required for tcp host mounts and for `--kernel host` guests; known + to transiently fail when the archive lags the runner image + (actions/runner-images#8080), so don't hard-fail the virtio path on + it; +- KVM enablement: the standard udev rule + (`KERNEL=="kvm", GROUP="kvm", MODE="0666"` + udevadm reload/trigger). + Since `run-under.sh` runs under `sudo -E` this is belt-and-braces; + still verify `/dev/kvm` exists and warn loudly when falling back to + TCG (a TCG git-annex run will blow the 2400 s budget -- treat TCG as + boot-smoke only, and let the cell fail fast with a clear message + rather than time out mutely). + +Timeouts: reuse the per-target values initially; virtio adds ~10-20 s +of boot, and 9p latency sits between ext4 and sync-NFS. Adjust from +evidence, not in advance. + +Rest of the standard checklist from "Adding a new backend" in the +README: `gen-readme-matrix.sh` regeneration, GOTCHAS "Backend settings" +section (mount options, security model, msize/cache defaults *as +measured*, guest kernel), README file-layout row, and +`shellcheck bin/ci/*.sh bin/eval-under*`. New files need no SPDX +headers (`bin/**`, `drafts/**`, `provision/**` are covered by +REUSE.toml's aggregate block). + +## Vagrant / local iteration + +- `provision/setup.sh`: add `qemu-system-x86 qemu-utils virtme-ng diod` + and `linux-modules-extra-$(uname -r)` (the cloud image's `-virtual` + kernel keeps 9p client modules there). Nested KVM already works: the + Vagrantfile sets `lv.nested = true` + `cpu_mode = "host-passthrough"`, + so `vng` inside the VM is hardware-accelerated. +- Vagrantfile, opt-in cross-check share: behind an env guard (say + `VAGRANT_9P_SHARE=1`), add a *second* synced folder of + `type: "9p"` at `/vagrant-9p` -- the genuine vagrant-libvirt article, + for validating that `eval-under-9p --transport virtio` reproduces the + semantics of the real thing (compare a pjdfstest run on both). The + default stays rsync; the existing comment explaining why remains + true for the *repo* share. Confirm vagrant-libvirt's current + `accessmode` default and owner/group options at implementation time + (docs were unreachable from the drafting environment). + +Local usage after landing: + + sudo bin/eval-under 9p --set-home -- bash -c 'cd "$HOME" && git annex test' + sudo bin/eval-under 9p --transport tcp --set-home -- git annex test + sudo bin/eval-under 9p --cache loose -- ... # the classic vagrant foot-gun + sudo -E bin/ci/run-under.sh 9p virtio-mapped pjdfstest + +## Phase 0: a probe, before any backend code + +One `workflow_dispatch` job (script in `bin/ci/`, per house rules -- +e.g. `bin/ci/probe-9p.sh`, kept afterwards as a doctor script), run on +both ubuntu-22.04 and ubuntu-24.04, reporting: + +1. `/dev/kvm` presence and usability (as root and as the runner user + with the udev rule); +2. whether `linux-modules-extra-$(uname -r)` installs, and whether + `modprobe 9p 9pnet 9pnet_tcp 9pnet_virtio` then succeeds on the + azure kernel (decides the tcp row's CI fate; the answer is genuinely + unknown -- packages.ubuntu.com contents search draws a blank); +3. `vng --run -- uname -a` boot smoke + a 5-line 9p + mount-and-touch inside the guest, and confirmation that a non-zero + guest exit propagates to the host. + +This converts every open risk below into a fact for the cost of one CI +run, before the backend script exists. + +## Risks and open questions + +| Risk | Exposure | Mitigation | +| --- | --- | --- | +| Azure kernel lacks 9p client modules | tcp row on hosted runners only | Probe decides; virtio row is immune (pinned `vng` kernel ships its own modules); tcp stays available locally/VM regardless. | +| `modules-extra` transiently uninstallable (runner image vs archive lag) | tcp row, `--kernel host` | Best-effort install; virtio row does not depend on it. | +| KVM on standard runners is unofficial | whole virtio row | Works today (udev rule; root via `sudo` regardless); probe verifies per-image; TCG fallback = fail fast with a clear message. | +| `vng` exit-status / stdout plumbing quirks | CI signal integrity | Verified explicitly in Phase 0 item 3. | +| virtiofsd availability for the vng *root* on jammy | only if 9p rows stay on ubuntu-22.04 | Prefer per-row `runs-on: ubuntu-24.04`; `vng --force-9p` for the rootfs is the fallback (slower, and amusingly turns even `/usr` into 9p). | +| Suite behavior differs guest-vs-host for non-fs reasons (loopback services, sockets) | git-annex target mostly | `git annex test` is local-only; add `--net user` to vng only if a target proves to need it. | + +## Sequencing + +1. **PR 1 -- probe.** `bin/ci/probe-9p.sh` + a tiny dispatch workflow; + record findings in the PR, then wire the answers into this plan. +2. **PR 2 -- the backend.** `bin/eval-under-9p` (virtio transport, + `mapped-xattr`), `9p)` cases in `run-under.sh` + + `install-backend.sh`, per-row `runs-on`, `9p-kernel` ref, + matrix row `virtio-mapped`, GOTCHAS settings section, README regen, + provision additions, `dump-failure-logs.sh` 9p case. +3. **PR 3 -- variants.** `virtio-passthrough` row; `tcp-diod` row if + the probe cleared it; Vagrantfile opt-in 9p share for + cross-validation. +4. **Later, own decisions:** `cache=loose` and msize variant rows + (GOTCHAS "Not yet covered" until then), and a sibling + `eval-under-virtiofs` backend -- the designated successor to 9p in + the same vagrant/QEMU role, nearly free once the vng plumbing + exists, and the natural control row for "is this 9p, or is this + any-VM-shared-fs?". From 318f73848c9e9cd1eca864405b4796b2f4f5d547 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 19:54:41 +0000 Subject: [PATCH 2/6] Add 9p backends: diod over TCP, and QEMU virtio-9p via virtme-ng Two new backends put 9p -- the filesystem behind Vagrant/libvirt "9p" synced folders, WSL2 drive shares, and QEMU -virtfs in general -- into the matrix, per the reviewed design in drafts/9p-backend-plan.md: - bin/eval-under-9p-tcp: diod (9P2000.L) exporting a fresh dir on 127.0.0.1, kernel v9fs mount -- the NFS backend's architectural sibling; no VM. diod runs unprivileged in its single-user mode by default; --run-as-root switches to the multi-user export for the privileged suites (the 9p analog of --no-root-squash). - bin/eval-under-9p-virtio: QEMU's virtfs server (-virtfs local,...) into a virtme-ng guest booted from the host's own rootfs, suite runs inside the guest; exit status and stdio come back over virtio-serial. Guest kernel pinned in CI (refs: 9p-kernel, handed over by run-under.sh -- the backend itself never reads matrix.yaml and defaults to the host kernel for local runs). Needs no host root for the mapped-xattr row. --shell drops into a live guest for debugging. Wiring: two matrix rows (9p-tcp/n/a, 9p-virtio/mapped) on ubuntu-24.04 via a new optional per-backend runs-on key (default ubuntu-22.04 keeps BeeGFS rows untouched); install_9p_tcp/_9p_virtio in install-backend.sh (incl. the KVM udev rule and a pre-warm boot that moves the ~180 MB pinned-kernel download outside the suite timeout); 9p cases in run-under.sh (needs-root wiring, VM timeout, --share-rw for git's test-results) and dump-failure-logs.sh; guest log teed to /var/log/eval-under-9p-virtio.log and added to the artifact list; GOTCHAS gains both settings tables and the semantics that make red 9p cells readable (mapped-xattr absorbs privileged ops; QEMU answers every TLOCK with success while diod takes whole-file flocks; diod caps msize at 64 KiB); provision installs the new deps. Validated end to end in a container: TCG boot of the pinned v6.8 mainline kernel, virtfs device visible (--disable-microvm), 9p mount + usability probe, argument fidelity through vng's double quoting layer (printf %q + a dq-escape pass), env trio on the mount, exit-status propagation. shellcheck-clean; README grid regenerated (28 cells); REUSE and codespell clean. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- .github/matrix.yaml | 30 ++ .github/workflows/test.yaml | 14 +- GOTCHAS.md | 92 ++++++ README.md | 26 +- bin/ci/dump-failure-logs.sh | 24 +- bin/ci/gen-readme-matrix.sh | 2 +- bin/ci/install-backend.sh | 76 ++++- bin/ci/matrix-json.sh | 11 +- bin/ci/matrix.sh | 10 +- bin/ci/render-badge.sh | 8 +- bin/ci/run-under.sh | 38 ++- bin/ci/update-status.py | 9 +- bin/eval-under-9p-tcp | 304 ++++++++++++++++++++ bin/eval-under-9p-virtio | 544 ++++++++++++++++++++++++++++++++++++ drafts/9p-backend-plan.md | 429 +++++++++------------------- provision/setup.sh | 12 + 16 files changed, 1295 insertions(+), 334 deletions(-) create mode 100755 bin/eval-under-9p-tcp create mode 100755 bin/eval-under-9p-virtio diff --git a/.github/matrix.yaml b/.github/matrix.yaml index 8efa5bb..681479a 100644 --- a/.github/matrix.yaml +++ b/.github/matrix.yaml @@ -34,6 +34,15 @@ src-dir: /opt/eval-under-src # did upstream add a test?). refs: git: v2.55.0 + # Guest kernel for the 9p-virtio backend's CI rows (vng downloads the + # Ubuntu mainline build; tag-shaped, leading "v" required). The v9fs + # *client* is the kernel, and its semantics move between releases + # (cache-mode rework in 6.4, netfs buffered writes in 6.8) -- an + # unpinned guest kernel would make a newly-red 9p cell ambiguous in + # exactly the way these pins exist to prevent. Local runs default to + # the host kernel instead; this pin is handed to the backend + # explicitly by bin/ci/run-under.sh. + 9p-kernel: v6.8 # pjd/pjdfstest carries exactly one tag upstream, "0.1" (2016), and it # no longer builds: major()/minor()/makedev() moved to # in glibc 2.28 and the tree compiles with -Werror, @@ -90,6 +99,12 @@ targets: # Row order of the README CI matrix. `version` is the literal "n/a" for # backends with nothing to pin (see bin/ci/install-backend.sh). +# +# `runs-on` (optional, default ubuntu-22.04) picks the runner image per +# row: BeeGFS DKMS constrains its rows to 22.04, while the 9p rows want +# 24.04, where virtme-ng/virtiofsd are packaged. Mixing images costs +# some cross-row comparability (different host kernel and tool +# versions) -- use it only when a backend actually needs it. backends: - backend: beegfs version: 7.4.6 @@ -106,3 +121,18 @@ backends: - backend: loop version: ext4 label: Loop ext4 + # v9fs client against diod (9P2000.L over TCP on localhost) -- the + # no-VM 9p row, architecturally the NFS backend's sibling. + - backend: 9p-tcp + version: n/a + label: 9p tcp (diod) + runs-on: ubuntu-24.04 + # v9fs client against QEMU's virtfs server over virtio, inside a + # virtme-ng guest -- the vagrant-libvirt "9p" synced-folder stack. + # "mapped" = security_model=mapped-xattr (vagrant accessmode + # "mapped"); vagrant's *default* (passthrough under an unprivileged + # QEMU) is a future row, see GOTCHAS.md "Not yet covered". + - backend: 9p-virtio + version: mapped + label: 9p virtio (mapped) + runs-on: ubuntu-24.04 diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index ad78515..b24c229 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -46,11 +46,12 @@ jobs: test: needs: matrix name: ${{ matrix.name }} - # ubuntu-22.04: kernel 5.15/6.5, within BeeGFS 7.4.x and 8.x DKMS - # support. ubuntu-24.04 hosted runners ship 6.17-azure which BeeGFS - # kernel modules cannot build against. NFS + loop backends don't - # need this, but sharing the runner OS keeps the matrix uniform. - runs-on: ubuntu-22.04 + # Per-cell runner image, from the backend row's optional `runs-on` + # in .github/matrix.yaml (default ubuntu-22.04: its kernel stays + # within BeeGFS DKMS support, while ubuntu-24.04 ships an azure + # kernel the BeeGFS modules cannot build against). The 9p rows + # override to ubuntu-24.04, where virtme-ng/virtiofsd are packaged. + runs-on: ${{ matrix.runs-on }} timeout-minutes: 60 strategy: fail-fast: false @@ -98,12 +99,13 @@ jobs: name: logs-${{ matrix.slug }} path: | /var/log/beegfs-* + /var/log/eval-under-9p-virtio.log /opt/eval-under-src/git/t/test-results/** if-no-files-found: ignore # One tiny artifact per cell, read back by the `badges` job. A # matrix job cannot contribute to a `needs.*.outputs` map, so this - # is how a fan-out reports 20 individual verdicts to a fan-in. + # is how a fan-out reports its per-cell verdicts to a fan-in. - name: Record cell result if: always() run: | diff --git a/GOTCHAS.md b/GOTCHAS.md index ff0883f..6072f6d 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -86,6 +86,80 @@ a kernel module built against the runner's kernel. | Client conf | `fixtures/beegfs/beegfs-client.conf.template` | Auth disabled, all daemons on `127.0.0.1`, non-default ports (8004-8008) so nothing collides with the runner. | | `sysMountSanityCheckMS` | `0` **on v8 only** | BeeGFS v8 dropped the standalone `beegfs-helperd` binary; with no helperd the sanity check cannot complete and the mount would hang. v7 keeps the check. | +### 9p tcp (`bin/eval-under-9p-tcp`) + +A fresh directory exported on `127.0.0.1` by diod (LLNL's 9P2000.L +server, Ubuntu universe) and mounted with the kernel v9fs client -- +architecturally the NFS backend's sibling. Same client code as the +virtio row below, different server. + +| Knob | Value | Why | +| --- | --- | --- | +| Server | `diod --foreground --no-auth --listen 127.0.0.1:5640 --export ` | Loopback throwaway export; munge auth has no place here. | +| Server identity | the *invoking user* (single-user mode), unless `--run-as-root` | diod's documented "simplest and safest" mode; also means the server itself needs no privilege. | +| Mount (default) | `-t 9p -o trans=tcp,port=5640,version=9p2000.L,aname=,uname=,access=` | diod's documented single-user pairing. `access=` restricts the mount to that uid -- root included, which is why the usability probe runs as the invoker. | +| Mount (`--run-as-root`) | `...,uname=root,access=client`, diod as root | diod's documented multi-user (NFS-like) pairing; the 9p analog of `--no-root-squash`, applied by `target_needs_root()`. | +| `msize` | not set -- kernel requests its default (128 KiB since 5.15), **diod caps at 64 KiB** | The `/proc/mounts` line the backend echoes records the effective value; results are relative to it. | +| `cache` | not set -- kernel default, which is **no caching in every kernel era** | `--cache loose` exists to reproduce the classic stale-read foot-gun deliberately. | + +**diod locking is whole-file.** diod implements 9P `Tlock` as BSD +`flock()` on the host file: byte-range `fcntl` locks collapse to +whole-file locks, and diod's own docs say distributed record locking +will deadlock ("test your use case!"). A locking red on this row is a +diod-server finding first -- compare with the virtio row, whose server +fakes locks differently (below), before blaming the client. + +### 9p virtio (`bin/eval-under-9p-virtio`) + +QEMU's virtfs server (`-virtfs local,...`) exporting a fresh host dir +into a virtme-ng guest booted from the host's own root filesystem; the +suite runs *inside the guest* on a `mount -t 9p -o trans=virtio` mount. +This is the vagrant-libvirt `type: "9p"` synced-folder stack. + +| Knob | Value | Why | +| --- | --- | --- | +| Server | QEMU `-virtfs local,path=,mount_tag=eval9p,security_model=mapped-xattr` (the `mapped` row) | vagrant-libvirt `accessmode: "mapped"`. Ownership/mode/devices are faked in `user.virtfs.*` xattrs on the host files. | +| Guest kernel (CI) | pinned in `.github/matrix.yaml` `refs: 9p-kernel` (`vng --run`, Ubuntu mainline build) | The v9fs *client is the kernel*: cache-mode rework landed in 6.4, netfs buffered writes in 6.8. Unpinned, a newly-red cell can't distinguish "filesystem regressed" from "client changed". Local runs default to the host kernel instead. | +| Guest machine | `vng --disable-microvm` | vng's microvm has no PCI bus stock guest kernels can enumerate; `-virtfs` is virtio-9p-**pci**, so without this the mount tag silently never appears. | +| Mount | `-t 9p -o trans=virtio,version=9p2000.L` (msize/cache unset = kernel defaults, as above; virtio transport caps msize at 512000) | The backend echoes the guest's `/proc/mounts` line and `uname -r` -- that pair is what a result is relative to. | +| Command identity | dropped to the invoking user via `runuser` in the guest; `--run-as-root` keeps root (`target_needs_root()` rows) | A vagrant user's synced folder is an unprivileged view; guest scripts otherwise run as root in virtme. | +| `writeout` | not set (QEMU default) | vagrant-libvirt sets `wrpolicy="immediate"`; `--writeout immediate` reproduces that when wanted. | + +**mapped-xattr absorbs privileged operations.** Under this security +model `chown` "succeeds" into an xattr, and `mknod` creates a host +*regular file* wearing a device-node xattr -- so pjdfstest's and +stress-ng's privileged assertions pass or fail against the *mapping +layer*, not a kernel. That is the measurement (it is exactly the +"returns success while doing the wrong thing" class stress-ng exists to +catch), but read those cells with this table in hand. Corollaries: unix +sockets and FIFOs work unprivileged; host-side the backing files read +as mode 0600/0700 owned by the QEMU user (`--keep` shows exactly that). + +**QEMU virtfs locking is a polite lie.** QEMU's 9p server answers every +`TLOCK` with success (single-client assumption); the guest kernel takes +the local VFS lock first, so locking is coherent *within* the guest and +invisible to the host. Locks therefore mostly "work" on this row where +diod's whole-file behavior differs -- a divergence between the two 9p +rows on locking tests is expected, not noise. + +**Other v9fs client facts red cells trace back to** (both 9p rows): +writable `MAP_SHARED` mmap fails `EINVAL` under the default no-cache +mode (read-only mmap -- git's main use -- works); `O_TMPFILE` is +unsupported (`EOPNOTSUPP`); there is no *remote*-change notification in +the protocol at all (inotify for the guest's own operations works +normally), which is also why `cache=loose` can serve stale data +indefinitely. + +**The guest is disposable; the log is not.** Everything the suite and +the guest console print is teed to `/var/log/eval-under-9p-virtio.log` +on the host (stage2 appends the guest `dmesg` tail on failure), because +by the time `dump-failure-logs.sh` runs, the guest -- and the backing +dir, via teardown -- are gone. vng quirk worth knowing: exit **255** +is also its "guest died before reporting" sentinel, so a suite exiting +255 is indistinguishable from a crash; and exit **124** under CI is the +backend's own `--vm-timeout` catching a boot/mount hang that the +in-guest per-target timeout cannot see. + ## Known-red cells A red cell here is a finding, not a bug report against this repo. These @@ -228,6 +302,24 @@ check the mount actually came up. `actimeo=0`), locking (`lock` vs `nolock`, and whether `rpc.statd` is even up), `sync` vs `async` on the export, and squashing. Each is a plausible row of its own. +- **The actual vagrant-libvirt 9p default.** vagrant-libvirt's default + `accessmode` is `passthrough`, and distro libvirt runs QEMU as an + unprivileged user -- so the out-of-the-box vagrant 9p share is + *passthrough served by a QEMU that cannot chown*, the very + configuration behind the classic "permission surprises". That is a + third variant, distinct from both the current `mapped` row and + passthrough-as-root. Reproduce it locally today: + `bin/eval-under-9p-virtio --security-model passthrough` run + *without* sudo. A `9p-virtio / passthrough` CI row (root QEMU) is the + cheap second row; the unprivileged flavour needs a small backend + tweak to skip sudo on that path. +- **9p cache/msize variants.** `--cache loose` (the vagrant stale-read + foot-gun) and a small-`msize` row are one matrix line each once the + base rows have settled. +- **virtiofs.** The designated successor to virtio-9p in the same + vagrant/QEMU role (`type: "virtiofs"`), and the natural control row + for "is this 9p, or any VM-shared filesystem?". Nearly free once the + vng plumbing exists: same guest, different device and server. - **Per-assertion breakdown** of the BeeGFS pjdfstest failures. - **A `Loop vfat` mask variant**, if we ever want to separate "vfat is not POSIX" from "vfat mounted with defaults is not POSIX". diff --git a/README.md b/README.md index 5101593..ed4e79f 100644 --- a/README.md +++ b/README.md @@ -30,10 +30,12 @@ itself is both backend- and suite-agnostic: new filesystems drop in as | NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | | Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | | Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | +| 9p tcp (diod) | [![9p tcp (diod) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-git-annex.svg)](https://con.github.io/eval-under/#9p-tcp-git-annex) | [![9p tcp (diod) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-git.svg)](https://con.github.io/eval-under/#9p-tcp-git) | [![9p tcp (diod) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-stress-ng.svg)](https://con.github.io/eval-under/#9p-tcp-stress-ng) | [![9p tcp (diod) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-pjdfstest.svg)](https://con.github.io/eval-under/#9p-tcp-pjdfstest) | +| 9p virtio (mapped) | [![9p virtio (mapped) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-git-annex.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-git-annex) | [![9p virtio (mapped) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-git.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-git) | [![9p virtio (mapped) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-stress-ng.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-stress-ng) | [![9p virtio (mapped) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-pjdfstest.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-pjdfstest) | Rows are **backends** (which filesystem the work happens on), columns -are **targets** (which suite is run on it). All 20 cells are one job +are **targets** (which suite is run on it). All 28 cells are one job matrix in [`.github/workflows/test.yaml`](.github/workflows/test.yaml), fanned out from [`.github/matrix.yaml`](.github/matrix.yaml) -- adding a filesystem or a suite is a data edit, not a code edit. @@ -145,6 +147,20 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \ # the fsync-heavy slow path) sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test' +# Under 9p served by diod over localhost TCP (the no-VM 9p; root is +# only used for the mount -- the server and the command run as you) +sudo bin/eval-under 9p-tcp --set-home -- bash -c 'cd "$HOME" && git annex test' + +# Under the real Vagrant/QEMU virtio-9p stack: the command runs INSIDE +# a virtme-ng guest booted from this host's own root filesystem. Needs +# no sudo at all (KVM access aside); --cache loose reproduces the +# classic vagrant stale-read foot-gun. +bin/eval-under 9p-virtio --set-home -- bash -c 'cd "$HOME" && git annex test' +bin/eval-under 9p-virtio --cache loose -- some-command + +# Poke around inside a live guest with the 9p mount up +bin/eval-under 9p-virtio --shell + # Skip teardown to poke around after a failure sudo bin/eval-under beegfs --set-home --keep -- some-failing-command @@ -166,6 +182,8 @@ the full flag / env-var / default table per backend. | `bin/eval-under-beegfs` | BeeGFS backend (containerised cluster + kernel client mount) | | `bin/eval-under-nfs` | NFS backend (localhost loopback export) | | `bin/eval-under-loop` | Loop-device backend (dd + losetup + mkfs. + mount) | +| `bin/eval-under-9p-tcp` | 9p backend, no VM: diod (9P2000.L) on localhost TCP + kernel v9fs mount | +| `bin/eval-under-9p-virtio` | 9p backend, the Vagrant/QEMU stack: QEMU -virtfs into a virtme-ng guest | | `fixtures/beegfs/docker-compose-v7.yml` | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host` | | `fixtures/beegfs/docker-compose-v8.yml` | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane) | | `fixtures/beegfs/beegfs-*.conf.template` | Minimal client + helperd confs for the throwaway cluster | @@ -179,7 +197,7 @@ the full flag / env-var / default table per backend. | `bin/ci/update-status.py` | Merges a run's per-cell results into the persistent `status.json` | | `bin/ci/render-report.py` | Renders `status.json` into the badge set + the report page | | `bin/ci/publish-status.sh` | Ties those together and pushes the site to `gh-pages` | -| `.github/workflows/test.yaml` | The whole matrix: one `matrix` job, 20 `test` cells, one `publish` job | +| `.github/workflows/test.yaml` | The whole matrix: one `matrix` job, 28 `test` cells, one `publish` job | | `drafts/git-annex-test-beegfs.yaml` | Copy-target workflow for `con/git-annex` (external PR target) | ## Local iteration (VM) @@ -198,6 +216,8 @@ sudo bin/eval-under beegfs --set-home -- bash -c ' ' sudo bin/eval-under nfs --set-home -- git annex test sudo bin/eval-under loop --fs vfat --set-home -- git annex test +sudo bin/eval-under 9p-tcp --set-home -- git annex test +bin/eval-under 9p-virtio --set-home -- git annex test # nested KVM is on # Or run a whole CI cell exactly as the runner would. install-target.sh # is the one-off prep (source builds land in $EVAL_UNDER_SRC_DIR, not on @@ -205,6 +225,8 @@ sudo bin/eval-under loop --fs vfat --set-home -- git annex test bin/ci/install-target.sh pjdfstest sudo -E bin/ci/run-under.sh loop ext4 pjdfstest sudo -E bin/ci/run-under.sh nfs n/a stress-ng +sudo -E bin/ci/run-under.sh 9p-tcp n/a pjdfstest +sudo -E bin/ci/run-under.sh 9p-virtio mapped stress-ng ``` Optional: install `act` in the VM to replay the GitHub workflow locally. diff --git a/bin/ci/dump-failure-logs.sh b/bin/ci/dump-failure-logs.sh index 6253b18..6317dcb 100755 --- a/bin/ci/dump-failure-logs.sh +++ b/bin/ci/dump-failure-logs.sh @@ -12,8 +12,12 @@ # bin/ci/dump-failure-logs.sh [target] # # backend side: -# beegfs: `docker compose logs` + dmesg-filtered-for-beegfs -# nfs/loop: full dmesg tail +# beegfs: `docker compose logs` + dmesg-filtered-for-beegfs +# nfs/loop: filtered dmesg tail +# 9p-tcp: filtered dmesg tail (the v9fs client logs there) +# 9p-virtio: tail of the guest console/suite log the backend tees to +# /var/log/eval-under-9p-virtio.log (guest dmesg included +# on failure; the guest itself is gone by now) # target side: # git: the failing assertions + output from t/test-results/*.out @@ -37,7 +41,7 @@ case "$BACKEND" in echo "=== dmesg (beegfs-tagged, last 50) ===" sudo dmesg | grep -i beegfs | tail -50 || true ;; - nfs|loop) + nfs|loop|9p-tcp) # Filtered rather than `dmesg | tail -100`: on a hosted runner the # last 100 kernel lines are almost entirely boot spam (hyperv, pci, # apparmor), which buries the failure in the job log. Warnings and @@ -47,9 +51,21 @@ case "$BACKEND" in sudo dmesg --level=emerg,alert,crit,err,warn 2>/dev/null | tail -40 || true echo "=== dmesg (mentioning $BACKEND/$VERSION, last 30) ===" sudo dmesg 2>/dev/null \ - | grep -iE "loop|nfs|${VERSION:-nomatch}" \ + | grep -iE "loop|nfs|9p|${VERSION:-nomatch}" \ | tail -30 || true ;; + 9p-virtio) + # The suite ran inside a virtme-ng guest that no longer exists; + # what survives is the console/suite log the backend tees on the + # host (stage2 appends the guest dmesg tail there on failure). + log="${EVAL_UNDER_9P_VIRTIO_LOG:-/var/log/eval-under-9p-virtio.log}" + echo "=== 9p-virtio guest log (last 120 lines of $log) ===" + if [ -r "$log" ]; then + tail -120 "$log" || true + else + echo "no log at $log (the guest never started?)" + fi + ;; *) echo "unknown backend: $BACKEND" >&2 ;; diff --git a/bin/ci/gen-readme-matrix.sh b/bin/ci/gen-readme-matrix.sh index ed3d5b8..94c6a62 100755 --- a/bin/ci/gen-readme-matrix.sh +++ b/bin/ci/gen-readme-matrix.sh @@ -72,7 +72,7 @@ table_md() { printf '\n' for cell in "${EVAL_UNDER_BACKENDS[@]}"; do - IFS='|' read -r backend version label <<< "$cell" + IFS='|' read -r backend version label _runs_on <<< "$cell" printf '| %s |' "$label" for target in "${EVAL_UNDER_TARGETS[@]}"; do slug="$(cell_slug "$backend" "$version" "$target")" diff --git a/bin/ci/install-backend.sh b/bin/ci/install-backend.sh index 6c5ee54..ae329c5 100755 --- a/bin/ci/install-backend.sh +++ b/bin/ci/install-backend.sh @@ -9,18 +9,27 @@ # usage: # bin/ci/install-backend.sh # -# backend = beegfs | nfs | loop -# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) -# for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) -# for nfs: literal "n/a" +# backend = beegfs | nfs | loop | 9p-tcp | 9p-virtio +# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) +# for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) +# for nfs: literal "n/a" +# for 9p-tcp: literal "n/a" +# for 9p-virtio: security-model row (mapped, passthrough) -- +# same packages either way # # Idempotent enough for CI re-runs; not a full package manager. set -euo pipefail export DEBIAN_FRONTEND=noninteractive -BACKEND="${1:?backend required (beegfs|nfs|loop)}" -VERSION="${2:?version required (BeeGFS version | loop fs name | 'n/a' for nfs)}" +here="$(cd "$(dirname "$0")" && pwd)" +# matrix.sh is a sourced library (pinned refs, e.g. the 9p guest +# kernel), resolved at runtime relative to $here. +# shellcheck source=bin/ci/matrix.sh disable=SC1091 +. "$here/matrix.sh" + +BACKEND="${1:?backend required (beegfs|nfs|loop|9p-tcp|9p-virtio)}" +VERSION="${2:?version required (BeeGFS version | loop fs name | 9p variant | 'n/a')}" # Give unattended-upgrades a moment on ubuntu-22.04 runners rather than # hard-failing on a dpkg lock. @@ -86,9 +95,56 @@ install_loop() { command -v "mkfs.$VERSION" } +install_9p_tcp() { + apt_update + apt_install diod + command -v diod + # 9p client modules ship in the kernel's base linux-modules package + # on generic, virtual and azure flavours alike -- no modules-extra + # needed. Load and prove the filesystem is registered now, so a + # missing module is an install-step error, not a mid-suite one. + sudo modprobe -a 9p 9pnet_fd + grep -w 9p /proc/filesystems +} + +install_9p_virtio() { + apt_update + # virtme-ng only Recommends qemu and Suggests virtiofsd, and this + # script installs with --no-install-recommends -- list everything + # explicitly. busybox-static: vng builds an initramfs for a pinned + # --run kernel and finds no busybox on the runner images otherwise. + # zstd: the mainline kernel debs ship .ko.zst modules and vng shells + # out to zstd for them (a guest that dies with vng's 255 sentinel + # after "zstd: not found" is this). Without virtiofsd, vng silently + # serves the rootfs over 9p instead (works, but slower and it + # shares the transport under test). + apt_install qemu-system-x86 qemu-utils virtme-ng virtiofsd busybox-static zstd + vng --version + + # KVM node permissions for the unprivileged prewarm below (the + # run-under.sh invocation is root via sudo -E and needs only the + # node to exist). Standard GitHub Linux runners have had KVM since + # 2024; this is the documented enablement. + echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ + | sudo tee /etc/udev/rules.d/99-kvm4all.rules >/dev/null + sudo udevadm control --reload-rules + sudo udevadm trigger --name-match=kvm + ls -l /dev/kvm + + # Pre-warm outside the suite's timeout: the pinned mainline kernel + # is a ~180 MB one-time download into ~/.cache/virtme-ng (the + # sudo -E run later resolves to the same HOME), and this doubles as + # a boot smoke test -- a vng/KVM problem fails the install step + # with its own logs instead of a red cell mid-matrix. + timeout 600 vng --run "$EVAL_UNDER_9P_KERNEL_REF" --disable-microvm \ + --memory 1G -- true +} + case "$BACKEND" in - beegfs) install_beegfs ;; - nfs) install_nfs ;; - loop) install_loop ;; - *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop)" >&2; exit 1 ;; + beegfs) install_beegfs ;; + nfs) install_nfs ;; + loop) install_loop ;; + 9p-tcp) install_9p_tcp ;; + 9p-virtio) install_9p_virtio ;; + *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop|9p-tcp|9p-virtio)" >&2; exit 1 ;; esac diff --git a/bin/ci/matrix-json.sh b/bin/ci/matrix-json.sh index 6c1ca13..bd48de6 100755 --- a/bin/ci/matrix-json.sh +++ b/bin/ci/matrix-json.sh @@ -14,10 +14,12 @@ # name job name, e.g. "BeeGFS 7.4.6 / git testsuite". Set as the # job's `name:` so the checks list reads like the README grid # instead of GitHub's default "test (beegfs, 7.4.6, git)". -# backend eval-under backend (beegfs | nfs | loop) +# backend eval-under backend (beegfs | nfs | loop | 9p-*) # version backend version, or "n/a" # target suite to run under it # slug filename-safe cell id, for artifact names +# runs-on runner image for this cell (per-backend in matrix.yaml, +# default ubuntu-22.04) # # usage: # bin/ci/matrix-json.sh # all cells @@ -34,9 +36,9 @@ here="$(cd "$(dirname "$0")" && pwd)" entries=() for cell in "${EVAL_UNDER_BACKENDS[@]}"; do - IFS='|' read -r backend version label <<< "$cell" + IFS='|' read -r backend version label runs_on <<< "$cell" for target in "${EVAL_UNDER_TARGETS[@]}"; do - entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") + entries+=("$backend|$version|$label|$runs_on|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") done done @@ -48,13 +50,14 @@ for line in sys.stdin: line = line.rstrip("\n") if not line: continue - backend, version, blabel, target, tlabel, slug = line.split("|") + backend, version, blabel, runs_on, target, tlabel, slug = line.split("|") include.append({ "name": "%s / %s" % (blabel, tlabel), "backend": backend, "version": version, "target": target, "slug": slug, + "runs-on": runs_on, }) if not include: diff --git a/bin/ci/matrix.sh b/bin/ci/matrix.sh index 9a2dd70..9320865 100755 --- a/bin/ci/matrix.sh +++ b/bin/ci/matrix.sh @@ -51,8 +51,13 @@ def q(v): out = [] out.append("EVAL_UNDER_TARGETS=(%s)" % " ".join(q(t["name"]) for t in targets)) +# Four-field tuple; consumers destructure with +# `IFS='|' read -r backend version label runs_on`. runs-on is optional +# in the YAML (default ubuntu-22.04) so existing rows stay untouched. out.append("EVAL_UNDER_BACKENDS=(%s)" % " ".join( - q("%s|%s|%s" % (b["backend"], b["version"], b["label"])) for b in backends)) + q("%s|%s|%s|%s" % (b["backend"], b["version"], b["label"], + b.get("runs-on", "ubuntu-22.04"))) + for b in backends)) out.append("declare -A _EU_LABEL=(%s)" % " ".join( "[%s]=%s" % (q(t["name"]), q(t["label"])) for t in targets)) @@ -70,6 +75,7 @@ out.append(": \"${EVAL_UNDER_REPO_SLUG:=%s}\"" % q(d["repo-slug"])) out.append(": \"${EVAL_UNDER_SRC_DIR:=%s}\"" % q(d["src-dir"])) out.append(": \"${EVAL_UNDER_GIT_REF:=%s}\"" % q(d["refs"]["git"])) out.append(": \"${EVAL_UNDER_PJDFSTEST_REF:=%s}\"" % q(d["refs"]["pjdfstest"])) +out.append(": \"${EVAL_UNDER_9P_KERNEL_REF:=%s}\"" % q(d["refs"]["9p-kernel"])) print("\n".join(out)) PYEOF @@ -83,7 +89,7 @@ PYEOF } export EVAL_UNDER_REPO_SLUG EVAL_UNDER_SRC_DIR -export EVAL_UNDER_GIT_REF EVAL_UNDER_PJDFSTEST_REF +export EVAL_UNDER_GIT_REF EVAL_UNDER_PJDFSTEST_REF EVAL_UNDER_9P_KERNEL_REF # Filename-safe identifier for a backend cell: "beegfs-7.4.6", "nfs", # "loop-vfat". diff --git a/bin/ci/render-badge.sh b/bin/ci/render-badge.sh index 4bf8ef2..db50a15 100755 --- a/bin/ci/render-badge.sh +++ b/bin/ci/render-badge.sh @@ -7,10 +7,10 @@ # Render one status badge as a self-contained SVG on stdout. # # We draw these ourselves rather than linking shields.io endpoint -# badges: the grid is 20 cells, so a README render would otherwise be -# 20 third-party requests, and the badges would go blank whenever that -# service is unreachable. An SVG committed to the `badges` branch has -# neither problem and is diffable. +# badges: the grid is dozens of cells, so a README render would +# otherwise be dozens of third-party requests, and the badges would go +# blank whenever that service is unreachable. An SVG committed to the +# `badges` branch has neither problem and is diffable. # # Single-segment on purpose: in the README grid the row and column # headers already name the cell, so a "BeeGFS 7.4.6 / git" prefix on diff --git a/bin/ci/run-under.sh b/bin/ci/run-under.sh index 8580891..5fc5659 100755 --- a/bin/ci/run-under.sh +++ b/bin/ci/run-under.sh @@ -13,10 +13,13 @@ # usage: # bin/ci/run-under.sh [target] # -# backend = beegfs | nfs | loop -# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) -# for loop: filesystem type (e.g. vfat, ext4) -# for nfs: literal "n/a" +# backend = beegfs | nfs | loop | 9p-tcp | 9p-virtio +# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) +# for loop: filesystem type (e.g. vfat, ext4) +# for nfs: literal "n/a" +# for 9p-tcp: literal "n/a" +# for 9p-virtio: QEMU virtfs security model row +# (mapped -> mapped-xattr, passthrough) # target = git-annex (default) | git | stress-ng | pjdfstest # # env overrides: @@ -59,6 +62,33 @@ case "$BACKEND" in # need an export that does not squash root, and need to keep # their privileges rather than being dropped to the invoker. target_needs_root "$TARGET" && opts=(--no-root-squash) ;; + 9p-tcp) + opts=() + # Same shape as NFS: the default is diod's single-user mode + # with the command dropped to the invoker; root-requiring + # suites need the multi-user export and their privileges. + target_needs_root "$TARGET" && opts=(--run-as-root) ;; + 9p-virtio) + # The version token is the QEMU virtfs security-model row. + case "$VERSION" in + mapped) secmodel=mapped-xattr ;; + passthrough) secmodel=passthrough ;; + *) echo "unknown 9p-virtio variant: $VERSION" >&2; exit 1 ;; + esac + # Pinned guest kernel: handed over explicitly, so the backend + # stays matrix-free and defaults to the host kernel locally. + # The VM timeout guards boot/mount hangs the in-guest + # per-target timeout cannot see; memory is CI-sized (the + # script's own default suits laptops). + opts=(--security-model "$secmodel" + --kernel "$EVAL_UNDER_9P_KERNEL_REF" + --memory 4G + --vm-timeout $((TIMEOUT + 300))) + # Suites that write results onto the runner's disk (git's + # t/test-results) need that dir shared read-write into the + # guest; everything else in the guest's overlay evaporates. + [ -d "$EVAL_UNDER_SRC_DIR" ] && opts+=(--share-rw "$EVAL_UNDER_SRC_DIR") + target_needs_root "$TARGET" && opts+=(--run-as-root) ;; *) echo "unknown backend: $BACKEND" >&2; exit 1 ;; esac diff --git a/bin/ci/update-status.py b/bin/ci/update-status.py index 40b77b1..316ea07 100755 --- a/bin/ci/update-status.py +++ b/bin/ci/update-status.py @@ -9,10 +9,11 @@ # # Why a persistent file rather than just reading the current run: a run # does not necessarily cover every cell. GitHub's "Re-run failed jobs" -# re-executes only the red ones, so a run's artifacts can describe 10 of -# 20 cells. Deriving the whole grid from one run would rewrite the other -# 10 badges to "unknown" and destroy good state. So results are merged, -# and a cell absent from this run keeps whatever it last reported. +# re-executes only the red ones, so a run's artifacts can describe a +# fraction of the grid. Deriving the whole grid from one run would +# rewrite every other badge to "unknown" and destroy good state. So +# results are merged, and a cell absent from this run keeps whatever it +# last reported. # # (This is a deliberate divergence from con/git-annex's update.py, which # sets absent tests to UNKNOWN. There, a client that stops reporting a diff --git a/bin/eval-under-9p-tcp b/bin/eval-under-9p-tcp new file mode 100755 index 0000000..1a0a334 --- /dev/null +++ b/bin/eval-under-9p-tcp @@ -0,0 +1,304 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-9p-tcp: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR (and +# optionally HOME) pointing at a 9p mount served over TCP by diod (LLNL's +# 9P2000.L server) on localhost -- same architecture as eval-under-nfs +# (local server + kernel client), different protocol and server. +# +# This exercises the kernel v9fs client against diod's server semantics. +# For the QEMU/virtio-9p stack that Vagrant synced folders actually use, +# see eval-under-9p-virtio; the v9fs client code is the same, the server +# (and its locking/caching quirks) is not. + +set -eu + +# Defaults (options override; env vars override defaults). +MNT="${EVAL_UNDER_MOUNT:-/mnt/9p}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +PORT="${EVAL_UNDER_9P_TCP_PORT:-5640}" +MSIZE="${EVAL_UNDER_9P_TCP_MSIZE:-}" +CACHE="${EVAL_UNDER_9P_TCP_CACHE:-}" +RUN_AS_ROOT="${EVAL_UNDER_9P_TCP_RUN_AS_ROOT:-0}" +STARTUP_WAIT="${EVAL_UNDER_9P_TCP_STARTUP_WAIT:-10}" + +usage() { + cat <<'EOF' +Usage: eval-under-9p-tcp [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at a 9p mount. A fresh backing directory is exported on +127.0.0.1 by diod (9P2000.L) and mounted via `mount -t 9p -o trans=tcp` +for the duration of the command. + +Requires the `diod` package (Ubuntu/Debian universe) and a kernel with +9p client support (CONFIG_9P_FS; in the stock `linux-modules` package +of Ubuntu generic/virtual/azure kernels). + +By default diod itself runs unprivileged AS THE INVOKING USER in its +documented single-user mode (mount options uname=,access=), +and the wrapped command runs as that user too -- root is used only for +the mount/umount pair. --run-as-root switches to diod's multi-user mode +(server as root, access=client) and keeps the wrapped command as root, +for suites that measure privileged behaviour. + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT /mnt/9p + Where to mount the export on the host. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Skip teardown; leave diod + mount up for debugging. + + --port P EVAL_UNDER_9P_TCP_PORT 5640 + TCP port for diod on 127.0.0.1 (unprivileged; not 564 so nothing + collides with a system-wide 9p service). + + --msize BYTES EVAL_UNDER_9P_TCP_MSIZE (unset) + Client request size. Unset = kernel default (128 KiB since 5.15) + -- which diod then negotiates down to its own 64 KiB cap. + + --cache MODE EVAL_UNDER_9P_TCP_CACHE (unset) + v9fs cache mode (none|loose|fscache|mmap|...; readahead/writeback + on kernels >= 6.4). Unset = kernel default, which is no caching in + every kernel era. `loose` is the classic stale-read foot-gun. + + --run-as-root EVAL_UNDER_9P_TCP_RUN_AS_ROOT (unset) + Run diod as root in multi-user mode (uname=root,access=client) + and keep the wrapped command running as root. Needed for suites + that measure privileged behaviour (pjdfstest, stress-ng's + chown/mknod stressors) -- the 9p analog of eval-under-nfs's + --no-root-squash. + + --startup-wait S EVAL_UNDER_9P_TCP_STARTUP_WAIT 10 + Seconds to wait for diod to bind its port before giving up. + + -h, --help + Print this help. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + +Exit status: the wrapped command's exit status. Teardown runs on any +exit (unless --keep). +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --port) PORT="$2"; shift 2 ;; + --msize) MSIZE="$2"; shift 2 ;; + --cache) CACHE="$2"; shift 2 ;; + --run-as-root) RUN_AS_ROOT=1; shift ;; + --startup-wait) STARTUP_WAIT="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ $# -gt 0 ] || { echo "no command given" >&2; usage >&2; exit 2; } + +if [ "$(id -u)" -ne 0 ]; then + command -v sudo >/dev/null || { + echo "must run as root (or have sudo available)" >&2; exit 2; } + SUDO=(sudo) +else + SUDO=() +fi + +# Identify the invoking (non-root) user: diod's single-user mode serves +# exactly one uid, and the wrapped command should run as the pre-sudo +# user, mirroring eval-under-nfs's root_squash default. +if [ -n "${SUDO_UID:-}" ]; then + INVOKER_UID="$SUDO_UID"; INVOKER_GID="${SUDO_GID:-$SUDO_UID}" + INVOKER_USER="${SUDO_USER:-$INVOKER_UID}" +else + INVOKER_UID="$(id -u)"; INVOKER_GID="$(id -g)" + INVOKER_USER="$(id -un)" +fi + +log() { printf '\nI: %s\n' "$*"; } + +ensure_9p_installed() { + command -v diod >/dev/null 2>&1 || { + echo "ERROR: diod not installed." >&2 + echo " Install with: apt install diod" >&2 + exit 3 + } + # 9p client modules live in the kernel's base linux-modules package + # (generic, virtual and azure flavours alike); modprobe is only needed + # because nothing else has loaded them yet. trans=tcp itself lives in + # 9pnet_fd (its own module since ~5.17), autoloaded on mount. + "${SUDO[@]}" modprobe -q 9p 2>/dev/null || true + grep -qw 9p /proc/filesystems || { + echo "ERROR: kernel has no 9p filesystem support (CONFIG_9P_FS)." >&2 + echo " On Ubuntu the modules ship in linux-modules-\$(uname -r);" >&2 + echo " a container kernel without them cannot run this backend." >&2 + exit 3 + } +} + +# Backing dir that gets exported (distinct from the mountpoint so we +# don't 9p-loop-back into ourselves). +ORIG="$(mktemp -u "${TMPDIR:-/tmp}/eval-under-9p-XXXXX").orig" +DIOD_LOG="$(mktemp "${TMPDIR:-/tmp}/eval-under-9p-diod-XXXXX.log")" +DIOD_PID="" + +start_diod() { + log "creating backing dir $ORIG (owned by uid=$INVOKER_UID gid=$INVOKER_GID)" + "${SUDO[@]}" mkdir -p "$ORIG" + "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$ORIG" + "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$DIOD_LOG" + + # --foreground so it stays our child (we background + kill it + # ourselves); --no-auth because munge has no place in a loopback + # throwaway export. + if [ "$RUN_AS_ROOT" != 0 ]; then + log "starting diod as root (multi-user): 127.0.0.1:$PORT <- $ORIG" + "${SUDO[@]}" diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + elif [ "$INVOKER_UID" != "$(id -u)" ]; then + log "starting diod as $INVOKER_USER (single-user): 127.0.0.1:$PORT <- $ORIG" + # The redirect deliberately happens in the current (root) shell, not + # under sudo -- the log was chown'd to the invoker above, and both + # sides can write it. + # shellcheck disable=SC2024 + sudo -u "$INVOKER_USER" diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + else + log "starting diod as $(id -un) (single-user): 127.0.0.1:$PORT <- $ORIG" + diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + fi + DIOD_PID=$! + + local waited=0 + while [ "$waited" -lt "$STARTUP_WAIT" ]; do + if ! kill -0 "$DIOD_PID" 2>/dev/null; then + break # died; report below + fi + if "${SUDO[@]}" ss -ltn 2>/dev/null | awk '{print $4}' | grep -qx "127.0.0.1:$PORT"; then + log "diod is listening (pid $DIOD_PID, after ${waited}s)" + return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + echo "ERROR: diod did not bind 127.0.0.1:$PORT within ${STARTUP_WAIT}s" >&2 + sed 's/^/ diod: /' "$DIOD_LOG" >&2 || true + exit 4 +} + +mount_9p() { + local mopts + # Option pairings are diod's documented ones: single-user mode serves + # exactly the invoking uid (uname=,access=); multi-user + # mode is the NFS-like uname=root,access=client. + if [ "$RUN_AS_ROOT" != 0 ]; then + mopts="trans=tcp,port=$PORT,version=9p2000.L,aname=$ORIG,uname=root,access=client" + else + mopts="trans=tcp,port=$PORT,version=9p2000.L,aname=$ORIG,uname=$INVOKER_USER,access=$INVOKER_UID" + fi + [ -n "$MSIZE" ] && mopts="$mopts,msize=$MSIZE" + [ -n "$CACHE" ] && mopts="$mopts,cache=$CACHE" + + log "mounting 9p at $MNT (-o $mopts)" + "${SUDO[@]}" mkdir -p "$MNT" + "${SUDO[@]}" mount -t 9p -o "$mopts" 127.0.0.1 "$MNT" + # The /proc/mounts line records the *effective* options -- diod caps + # msize at 64 KiB whatever the client asked for, and GOTCHAS.md wants + # settings as measured, not as requested. + grep -F " $MNT " /proc/mounts | sed 's/^/I: /' || true +} + +# mount(2) returning is not the same as the export being usable; probe +# with a real create + write + read-back, as the invoker when access= is +# restricted to that uid (root is not exempt from v9fs access checks). +wait_for_mount_usable() { + local probe="$MNT/.eval-under-ready.$$" + local as_user=() + if [ "$RUN_AS_ROOT" = 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + as_user=(sudo -u "$INVOKER_USER") + fi + local waited=0 err="" + while [ "$waited" -lt 10 ]; do + if err=$("${as_user[@]}" dd if=/dev/urandom of="$probe" bs=4k count=1 2>&1) \ + && err=$("${as_user[@]}" dd if="$probe" of=/dev/null bs=4k count=1 2>&1); then + "${as_user[@]}" rm -f "$probe" 2>/dev/null || true + log "mount is usable (after ${waited}s)" + return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + echo "ERROR: 9p mounted at $MNT but unusable after 10s" >&2 + echo " last probe error: $err" >&2 + sed 's/^/ diod: /' "$DIOD_LOG" >&2 || true + "${SUDO[@]}" dmesg 2>/dev/null | grep -i 9p | tail -20 >&2 || true + exit 6 +} + +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving diod (pid $DIOD_PID) + mount up (exit=$rc)" + echo "I: diod log: $DIOD_LOG" + return "$rc" + fi + log "teardown" + "${SUDO[@]}" umount "$MNT" 2>/dev/null || true + if [ -n "$DIOD_PID" ]; then + "${SUDO[@]}" kill "$DIOD_PID" 2>/dev/null || true + fi + "${SUDO[@]}" rm -rf "$MNT" "$ORIG" "$DIOD_LOG" 2>/dev/null || true + return "$rc" +} +trap teardown EXIT + +ensure_9p_installed +start_diod +mount_9p +wait_for_mount_usable + +RUN_HOME="$HOME" +if [ "$SET_HOME" = 1 ]; then + RUN_HOME="$MNT/home" + # mkdir as the invoker -- under access= a root mkdir would be + # rejected by the client, same shape as NFS root_squash. + if [ "$RUN_AS_ROOT" = 0 ] \ + && [ "$INVOKER_UID" != 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + sudo -u "$INVOKER_USER" mkdir -p "$RUN_HOME" + else + mkdir -p "$RUN_HOME" + fi +fi + +# Drop back to the invoking user for the wrapped command (single-user +# diod serves only that uid anyway); --run-as-root keeps privileges to +# match its multi-user export. Mirrors eval-under-nfs. +if [ "$RUN_AS_ROOT" = 0 ] \ + && [ "$INVOKER_UID" != 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + log "running (as uid=$INVOKER_UID): $*" + sudo -u "$INVOKER_USER" -E \ + env HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" \ + "$@" +else + log "running: $*" + HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" "$@" +fi diff --git a/bin/eval-under-9p-virtio b/bin/eval-under-9p-virtio new file mode 100755 index 0000000..bf3e67b --- /dev/null +++ b/bin/eval-under-9p-virtio @@ -0,0 +1,544 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-9p-virtio: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR +# (and optionally HOME) pointing at a virtio-9p mount -- the actual stack +# behind Vagrant/libvirt "9p" synced folders, WSL2 drive shares, and +# QEMU -virtfs in general: QEMU's virtfs server on the host, the kernel +# v9fs client in a guest, virtio in between. +# +# The guest comes from virtme-ng (vng): it boots a QEMU guest whose root +# filesystem is a read-only view of the host's own root (with tmpfs +# overlays), so every tool installed on the host -- git, git-annex, +# prove, the bin/ci target scripts -- exists in the guest unchanged. +# This script adds one more -virtfs device exporting a fresh scratch +# dir, re-invokes itself inside the guest (--guest-stage2) to mount it +# and run the wrapped command there, and propagates the exit status +# back out (vng carries it over a dedicated virtio-serial channel). +# +# Notes that shape the implementation: +# - vng's rootfs is read-only outside its overlay set (/etc /lib /home +# /opt /srv /usr /var, plus a fresh tmpfs /tmp), so the *guest* +# mountpoint defaults to /tmp/eval-under-9p and --mount-point here +# names a guest path. Writes to overlay paths evaporate with the +# guest; use --share-rw for host dirs that must receive writes. +# - vng's microvm machine has no PCI bus the stock Ubuntu/mainline +# guest kernels can enumerate, so --disable-microvm is passed to +# keep -virtfs (virtio-9p-pci) visible. +# - guest scripts run as root by default; the wrapped command is +# dropped to the invoking user inside the guest (runuser) unless +# --run-as-root, mirroring eval-under-nfs. +# - unlike the other backends this one needs NO root on the host for +# security models other than passthrough-as-root: QEMU, vng and the +# mount (which happens inside the guest) all run unprivileged. + +set -eu + +here="$(cd "$(dirname "$0")" && pwd)" +SELF="$here/$(basename "$0")" + +# Defaults (options override; env vars override defaults). +MNT="${EVAL_UNDER_MOUNT:-/tmp/eval-under-9p}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +SECMODEL="${EVAL_UNDER_9P_VIRTIO_SECURITY_MODEL:-mapped-xattr}" +KERNEL="${EVAL_UNDER_9P_VIRTIO_KERNEL:-host}" +MEMORY="${EVAL_UNDER_9P_VIRTIO_MEMORY:-2G}" +CPUS="${EVAL_UNDER_9P_VIRTIO_CPUS:-$(nproc)}" +MSIZE="${EVAL_UNDER_9P_VIRTIO_MSIZE:-}" +CACHE="${EVAL_UNDER_9P_VIRTIO_CACHE:-}" +WRITEOUT="${EVAL_UNDER_9P_VIRTIO_WRITEOUT:-}" +RUN_AS_ROOT="${EVAL_UNDER_9P_VIRTIO_RUN_AS_ROOT:-0}" +VM_TIMEOUT="${EVAL_UNDER_9P_VIRTIO_VM_TIMEOUT:-0}" +ALLOW_TCG="${EVAL_UNDER_9P_VIRTIO_ALLOW_TCG:-0}" +LOG="${EVAL_UNDER_9P_VIRTIO_LOG:-}" +DEBUG_BOOT="${EVAL_UNDER_9P_VIRTIO_DEBUG_BOOT:-0}" +SHELL_MODE=0 +TAG="eval9p" + +# Space-separated initial value from the environment; --share-rw appends. +read -r -a SHARE_RW <<< "${EVAL_UNDER_9P_VIRTIO_SHARE_RW:-}" + +usage() { + cat <<'EOF' +Usage: eval-under-9p-virtio [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at a virtio-9p mount: a fresh scratch dir on the host is +exported by QEMU's virtfs server (-virtfs local,...) into a virtme-ng +guest booted from the host's own root filesystem, mounted there with +`mount -t 9p -o trans=virtio`, and CMD runs INSIDE that guest. This is +the stack Vagrant/libvirt "9p" synced folders use. + +Requires: virtme-ng (vng), qemu-system-x86; busybox for a pinned +--kernel (package busybox-static); /dev/kvm strongly recommended. +Install on Ubuntu 24.04: apt install virtme-ng virtiofsd busybox-static +(22.04 has no virtme-ng package: pipx install virtme-ng). + +Unlike the other backends, no host root is needed except for +--security-model passthrough served by a root QEMU (run the whole +script under sudo for that). CI does run it under sudo -E; that is +what makes its passthrough row "passthrough as root". + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT /tmp/eval-under-9p + GUEST path to mount the export on. Must be creatable in the + guest: vng's rootfs is read-only outside /etc /lib /home /opt + /srv /usr /var and a fresh tmpfs /tmp -- the default lives there + on purpose. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Keep the host-side backing dir + log after exit and print how to + relaunch. The guest itself always ends with the command; use + --shell for an interactive look INSIDE a live guest. Note that + under mapped-xattr the kept files read host-side as mode 0600/0700 + owned by the QEMU user, with the real metadata in user.virtfs.* + xattrs. + + --shell (unset) + Instead of CMD, run an interactive bash inside the guest with the + mount already up (best-effort: the console is a virtio-serial + channel, so no job control). The morning-after-a-red-cell tool. + + --security-model M EVAL_UNDER_9P_VIRTIO_SECURITY_MODEL mapped-xattr + QEMU virtfs server model: mapped-xattr (ownership/mode/devices + faked in user.virtfs.* xattrs -- vagrant-libvirt accessmode + "mapped"), mapped-file, passthrough (real uids; wants a root + QEMU -- run this script under sudo), none (passthrough that + ignores chown failures -- vagrant-libvirt "squash"). + NOTE vagrant-libvirt's *default* accessmode is passthrough under + an unprivileged QEMU; reproduce that by running this script + unprivileged with --security-model passthrough. + + --kernel VER EVAL_UNDER_9P_VIRTIO_KERNEL host + Guest kernel. "host" boots the host's own kernel and modules (no + download; /boot/vmlinuz-* must be readable -- Ubuntu ships it + 0600, so chmod +r it or run under sudo). A version tag like + "v6.8" makes vng download that Ubuntu mainline build (~180 MB + once, cached in ~/.cache/virtme-ng) -- what CI uses so the v9fs + client is pinned. + + --memory SIZE EVAL_UNDER_9P_VIRTIO_MEMORY 2G + Guest RAM (qemu -m syntax). Overlay writes live in guest RAM, so + a suite that writes a lot outside the mount needs headroom. + + --cpus N EVAL_UNDER_9P_VIRTIO_CPUS nproc + Guest vCPUs. + + --msize BYTES EVAL_UNDER_9P_VIRTIO_MSIZE (unset) + Client request size. Unset = kernel default (128 KiB since 5.15; + virtio transport caps at 512000). + + --cache MODE EVAL_UNDER_9P_VIRTIO_CACHE (unset) + v9fs cache mode. Unset = kernel default (no caching, every + kernel era). `loose` is the classic vagrant stale-read foot-gun. + + --writeout immediate EVAL_UNDER_9P_VIRTIO_WRITEOUT (unset) + Pass writeout=immediate to the virtfs server -- what + vagrant-libvirt sets (wrpolicy="immediate"). Unset = QEMU default. + + --run-as-root EVAL_UNDER_9P_VIRTIO_RUN_AS_ROOT (unset) + Keep the wrapped command running as root inside the guest + (default drops to the invoking user with runuser). The 9p analog + of eval-under-nfs's --no-root-squash; needed by pjdfstest and + stress-ng's chown/mknod stressors. Under mapped-xattr their + privileged ops "succeed" into xattr metadata -- that is the + measurement, see GOTCHAS.md. + + --share-rw PATH EVAL_UNDER_9P_VIRTIO_SHARE_RW (none) + Host directory to share read-write into the guest at the same + path (vng --rwdir). Repeatable; env var is space-separated. CI + passes the target-suite source dir so test-results/ written by + the suite reach the host for artifact upload. + + --vm-timeout S EVAL_UNDER_9P_VIRTIO_VM_TIMEOUT 0 (off) + Host-side timeout around the whole guest. The per-target timeout + CI wraps around CMD runs *inside* the guest and cannot catch a + hung boot; this one can. Exit 124 = the VM was killed. + + --allow-tcg EVAL_UNDER_9P_VIRTIO_ALLOW_TCG (unset) + Proceed without /dev/kvm (pure emulation, an order of magnitude + slower). Default is to fail fast with a clear message instead of + mutely blowing every suite budget. + + --log FILE EVAL_UNDER_9P_VIRTIO_LOG /var/log/eval-under-9p-virtio.log + Host file receiving a copy of all guest console/suite output + (falls back to a mktemp path when unwritable, e.g. unprivileged + local runs). What bin/ci/dump-failure-logs.sh and the CI artifact + pick up after teardown. + + --debug-boot EVAL_UNDER_9P_VIRTIO_DEBUG_BOOT (unset) + Add vng --show-boot-console --verbose: boot messages are + suppressed by default, so a boot-time failure is otherwise + silent. + + -h, --help + Print this help. + +Internal (visible so a stuck run can be debugged by hand): + + --guest-stage2 ... + The half of this script that runs inside the guest, as guest + root: modprobe + mount + usability probe + user drop + exec CMD. + The host half passes every argument shell-quoted through vng's + single command-string channel. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = (a guest path here) + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + PATH = the host-side PATH, passed through + +Exit status: the wrapped command's, carried out of the guest by vng. +Caveats: 124 with --vm-timeout means the VM itself was killed; 255 is +also vng's own "guest died before reporting" sentinel, so a command +exiting 255 is indistinguishable from a crash. No network inside the +guest. Teardown removes the host backing dir (unless --keep). +EOF +} + +log() { printf '\nI: %s\n' "$*"; } + +# --------------------------------------------------------------------------- +# Guest half. Runs first so none of the host-side setup below executes in +# the guest. Everything here runs as guest root (vng script default). +# --------------------------------------------------------------------------- +if [ "${1:-}" = "--guest-stage2" ]; then + shift + G_TAG="" G_MNT="" G_MOPTS="" G_PATH="$PATH" G_DROP_TO="" G_SET_HOME=0 G_SHELL=0 + while [ $# -gt 0 ]; do + case "$1" in + --tag) G_TAG="$2"; shift 2 ;; + --mount-point) G_MNT="$2"; shift 2 ;; + --mount-opts) G_MOPTS="$2"; shift 2 ;; + --path) G_PATH="$2"; shift 2 ;; + --drop-to) G_DROP_TO="$2"; shift 2 ;; + --set-home) G_SET_HOME=1; shift ;; + --shell) G_SHELL=1; shift ;; + --) shift; break ;; + *) echo "guest-stage2: unknown arg: $1" >&2; exit 2 ;; + esac + done + if [ -z "$G_TAG" ] || [ -z "$G_MNT" ] || [ -z "$G_MOPTS" ]; then + echo "guest-stage2: --tag/--mount-point/--mount-opts required" >&2 + exit 2 + fi + + guest_dmesg_tail() { + echo "=== guest dmesg (9p/virtio, last 30) ===" >&2 + dmesg 2>/dev/null | grep -iE '9p|virtio' | tail -30 >&2 || true + } + + # Module load: a pinned mainline kernel has these preloaded by virtme's + # MODALIASES, the host kernel usually not. Best-effort; the mount is + # the real test. + modprobe -qa 9p 9pnet_virtio 2>/dev/null || true + + mkdir -p "$G_MNT" || { echo "guest-stage2: cannot mkdir $G_MNT (read-only rootfs path? see --mount-point in --help)" >&2; exit 7; } + if ! mount -t 9p -o "$G_MOPTS" "$G_TAG" "$G_MNT"; then + echo "ERROR: mount -t 9p -o $G_MOPTS $G_TAG $G_MNT failed in the guest" >&2 + echo " (a missing mount tag usually means the -virtfs device is not visible)" >&2 + guest_dmesg_tail + exit 7 + fi + + # Probe with a real create + write + read-back before handing over. + probe="$G_MNT/.eval-under-ready.$$" + if ! dd if=/dev/urandom of="$probe" bs=4k count=1 2>/dev/null \ + || ! dd if="$probe" of=/dev/null bs=4k count=1 2>/dev/null; then + echo "ERROR: 9p mounted at $G_MNT but a create+write+read probe failed" >&2 + guest_dmesg_tail + exit 6 + fi + rm -f "$probe" 2>/dev/null || true + + echo "I: guest kernel $(uname -r)" + grep -F " $G_MNT " /proc/mounts | sed 's/^/I: /' || true + + if [ -n "$G_DROP_TO" ]; then + chown "$G_DROP_TO" "$G_MNT" 2>/dev/null || true + fi + + RUN_HOME="${HOME:-/root}" + if [ "$G_SET_HOME" = 1 ]; then + RUN_HOME="$G_MNT/home" + if [ -n "$G_DROP_TO" ]; then + runuser -u "$G_DROP_TO" -- mkdir -p "$RUN_HOME" + else + mkdir -p "$RUN_HOME" + fi + fi + + if [ "$G_SHELL" = 1 ]; then + echo "I: interactive guest shell; the mount is at $G_MNT (exit to end the VM)" + export HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" + if [ -n "$G_DROP_TO" ]; then + exec runuser -u "$G_DROP_TO" -- bash -i + fi + exec bash -i + fi + + echo "I: running in guest${G_DROP_TO:+ (as $G_DROP_TO)}: $*" + rc=0 + if [ -n "$G_DROP_TO" ]; then + runuser -u "$G_DROP_TO" -- \ + env HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" \ + "$@" || rc=$? + else + env HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" \ + "$@" || rc=$? + fi + if [ "$rc" -ne 0 ]; then + guest_dmesg_tail + fi + exit "$rc" +fi + +# --------------------------------------------------------------------------- +# Host half. +# --------------------------------------------------------------------------- +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --shell) SHELL_MODE=1; shift ;; + --security-model) SECMODEL="$2"; shift 2 ;; + --kernel) KERNEL="$2"; shift 2 ;; + --memory) MEMORY="$2"; shift 2 ;; + --cpus) CPUS="$2"; shift 2 ;; + --msize) MSIZE="$2"; shift 2 ;; + --cache) CACHE="$2"; shift 2 ;; + --writeout) WRITEOUT="$2"; shift 2 ;; + --run-as-root) RUN_AS_ROOT=1; shift ;; + --share-rw) SHARE_RW+=("$2"); shift 2 ;; + --vm-timeout) VM_TIMEOUT="$2"; shift 2 ;; + --allow-tcg) ALLOW_TCG=1; shift ;; + --log) LOG="$2"; shift 2 ;; + --debug-boot) DEBUG_BOOT=1; shift ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [ "$SHELL_MODE" = 0 ] && [ $# -eq 0 ]; then + echo "no command given" >&2; usage >&2; exit 2 +fi + +case "$SECMODEL" in + mapped|mapped-xattr|mapped-file|passthrough|none) ;; + *) echo "unknown --security-model: $SECMODEL (mapped-xattr|mapped-file|passthrough|none)" >&2; exit 2 ;; +esac + +# The user the wrapped command should run as inside the guest (the guest +# sees the host's /etc/passwd through the shared rootfs). +if [ -n "${SUDO_USER:-}" ]; then + INVOKER_USER="$SUDO_USER"; INVOKER_UID="${SUDO_UID:-0}" +else + INVOKER_USER="$(id -un)"; INVOKER_UID="$(id -u)" +fi +[ "$INVOKER_UID" = 0 ] && RUN_AS_ROOT=1 # dropping root to root is a no-op + +ensure_virtio_installed() { + command -v vng >/dev/null 2>&1 || { + echo "ERROR: virtme-ng (vng) not installed." >&2 + echo " Install with: apt install virtme-ng (Ubuntu 24.04+)" >&2 + echo " or: pipx install virtme-ng (22.04 and other distros)" >&2 + exit 3 + } + command -v qemu-system-x86_64 >/dev/null 2>&1 || { + echo "ERROR: qemu-system-x86_64 not installed." >&2 + echo " Install with: apt install qemu-system-x86" >&2 + exit 3 + } + if [ "$KERNEL" != host ] && ! command -v busybox >/dev/null 2>&1; then + echo "ERROR: a pinned --kernel needs an initramfs, which vng builds with busybox." >&2 + echo " Install with: apt install busybox-static" >&2 + exit 3 + fi + if [ "$KERNEL" != host ] && ! command -v zstd >/dev/null 2>&1; then + echo "ERROR: mainline kernel debs ship .ko.zst modules; vng needs zstd for them" >&2 + echo " (the failure mode without it is an opaque guest death, exit 255)." >&2 + echo " Install with: apt install zstd" >&2 + exit 3 + fi + if [ "$KERNEL" = host ]; then + local vmlinuz + vmlinuz="/boot/vmlinuz-$(uname -r)" + if [ -e "$vmlinuz" ] && [ ! -r "$vmlinuz" ]; then + echo "ERROR: $vmlinuz is not readable (Ubuntu ships it 0600)." >&2 + echo " Run under sudo, chmod +r it, or use --kernel v instead." >&2 + exit 3 + fi + fi + if [ ! -e /dev/kvm ]; then + if [ "$ALLOW_TCG" != 0 ]; then + echo "W: no /dev/kvm -- continuing under TCG emulation (SLOW)" >&2 + else + echo "ERROR: no /dev/kvm. A TCG-emulated guest is an order of magnitude" >&2 + echo " slower and blows every suite budget; pass --allow-tcg to insist." >&2 + exit 3 + fi + fi + if ! command -v virtiofsd >/dev/null 2>&1 \ + && [ ! -x /usr/lib/virtiofsd ] && [ ! -x /usr/libexec/virtiofsd ] \ + && [ ! -x /usr/lib/qemu/virtiofsd ]; then + echo "W: no virtiofsd found; vng will fall back to a 9p rootfs (slower boot," >&2 + echo " and the root filesystem shares the transport under test)." >&2 + fi + # The guest boots this host's own rootfs, so guest-side plumbing + # depends on host packages: virtme-init needs udevd to surface the + # virtio-serial script ports (without it the guest powers off with + # "cannot find script I/O ports" and vng reports its 255 sentinel). + # Present on any normal Ubuntu install; missing in minimal containers. + if ! command -v udevadm >/dev/null 2>&1; then + echo "W: no udev on this host -- the guest will likely fail with" >&2 + echo " 'cannot find script I/O ports'. Install with: apt install udev" >&2 + fi +} + +# Host-side backing dir that QEMU exports. Its path is embedded in a +# comma-separated -virtfs value and in a space-joined qemu-opts string, +# so refuse the characters that would silently corrupt those. +ORIG="$(mktemp -d "${TMPDIR:-/tmp}/eval-under-9p-XXXXX.orig")" +case "$ORIG" in + *[,\ ]*) echo "ERROR: backing dir path '$ORIG' contains a comma or space (set TMPDIR elsewhere)" >&2; exit 2 ;; +esac + +if [ -z "$LOG" ]; then + LOG=/var/log/eval-under-9p-virtio.log + if ! touch "$LOG" 2>/dev/null; then + LOG="$(mktemp "${TMPDIR:-/tmp}/eval-under-9p-virtio-XXXXX.log")" + fi +fi + +MOPTS="trans=virtio,version=9p2000.L" +[ -n "$MSIZE" ] && MOPTS="$MOPTS,msize=$MSIZE" +[ -n "$CACHE" ] && MOPTS="$MOPTS,cache=$CACHE" + +VIRTFS="local,path=$ORIG,mount_tag=$TAG,security_model=$SECMODEL,id=$TAG" +[ -n "$WRITEOUT" ] && VIRTFS="$VIRTFS,writeout=$WRITEOUT" + +build_vng_cmd() { + VNG_CMD=(vng) + [ "$KERNEL" != host ] && VNG_CMD+=(--run "$KERNEL") + # microvm has no PCI bus the stock guest kernels can enumerate; the + # -virtfs device below is virtio-9p-pci, so force a PCI-ful machine. + VNG_CMD+=(--disable-microvm --memory "$MEMORY" --cpus "$CPUS") + local d + for d in ${SHARE_RW[@]+"${SHARE_RW[@]}"}; do + VNG_CMD+=(--rwdir "$d") + done + # =-joined: the value starts with a dash and argparse would otherwise + # read it as the next option. + VNG_CMD+=("--qemu-opts=-virtfs $VIRTFS") + if [ "$DEBUG_BOOT" != 0 ]; then + # vng 1.22 has no separate boot-console flag; --verbose shows the + # boot console and the underlying virtme-run line. + VNG_CMD+=(--verbose) + fi +} + +# The command travels through exactly two shell layers, so quoting is +# OUR job, done in two matching passes: +# +# layer 1: vng embeds the joined words after `--` verbatim inside +# double quotes in a `virtme-run ... --script-sh "..."` +# line and runs it via `sh -c` (check_call(shell=True)) -- +# a POSIX double-quote context, which consumes one level of +# \\ \" \$ \` escapes; +# layer 2: virtme-run base64s the surviving string through the kernel +# cmdline and the guest executes it as a *bash* script. +# +# So: printf %q makes each argument safe for the guest bash (layer 2), +# and dq_escape pre-compensates the double-quote context (layer 1) so +# the %q text arrives in the guest byte-identical. Without the second +# pass, a wrapped `bash -c 'cd "$HOME" && ...'` would have $HOME +# expanded one layer early, as guest root. +dq_escape() { + local s="$1" + s=${s//\\/\\\\} + s=${s//\"/\\\"} + s=${s//\$/\\\$} + s=${s//\`/\\\`} + printf '%s' "$s" +} + +build_stage2_string() { + local args=("$SELF" --guest-stage2 + --tag "$TAG" --mount-point "$MNT" --mount-opts "$MOPTS" + --path "$PATH") + [ "$SET_HOME" = 1 ] && args+=(--set-home) + [ "$RUN_AS_ROOT" = 0 ] && args+=(--drop-to "$INVOKER_USER") + [ "$SHELL_MODE" = 1 ] && args+=(--shell) + args+=(--) + local a + STAGE2_STR="" + for a in "${args[@]}" "$@"; do + STAGE2_STR+="$(dq_escape "$(printf '%q' "$a")") " + done +} + +# Invoked only via the EXIT trap; shellcheck's reachability analysis +# cannot see that and calls the body unreachable (same annotation as +# matrix.sh carries for its trap/return fallback). +# shellcheck disable=SC2317 +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving backing dir + log (exit=$rc)" + echo "I: backing dir: $ORIG (mapped-xattr metadata lives in user.virtfs.* xattrs)" + echo "I: guest log: $LOG" + echo "I: relaunch interactively with: $0 --shell ${SECMODEL:+--security-model $SECMODEL}" + return "$rc" + fi + rm -rf "$ORIG" 2>/dev/null || true + # $LOG is deliberately kept: it is what bin/ci/dump-failure-logs.sh + # and the CI artifact read after this process is gone. + return "$rc" +} +trap teardown EXIT + +ensure_virtio_installed +build_vng_cmd +build_stage2_string "$@" + +log "guest: kernel=$KERNEL mem=$MEMORY cpus=$CPUS log=$LOG" +if [ "${#SHARE_RW[@]}" -gt 0 ]; then + log "rw dirs shared into guest: ${SHARE_RW[*]}" +fi +log "virtfs: $VIRTFS" +log "mount (in guest): -t 9p -o $MOPTS $TAG $MNT" + +RUNNER=("${VNG_CMD[@]}" -- "$STAGE2_STR") +if [ "$VM_TIMEOUT" != 0 ]; then + RUNNER=(timeout --kill-after=30 "$VM_TIMEOUT" "${RUNNER[@]}") +fi + +rc=0 +if [ "$SHELL_MODE" = 1 ]; then + # Interactive: no tee (it would steal the terminal). + "${RUNNER[@]}" || rc=$? +else + "${RUNNER[@]}" 2>&1 | tee -a "$LOG" || true + rc="${PIPESTATUS[0]}" +fi + +if [ "$rc" -eq 124 ] && [ "$VM_TIMEOUT" != 0 ]; then + echo "ERROR: guest exceeded --vm-timeout ${VM_TIMEOUT}s and was killed (boot or mount hang?)" >&2 +elif [ "$rc" -eq 255 ]; then + echo "W: exit 255 -- either the wrapped command exited 255, or the guest died" >&2 + echo " before reporting (vng's sentinel). Re-run with --debug-boot to see boot." >&2 +fi +exit "$rc" diff --git a/drafts/9p-backend-plan.md b/drafts/9p-backend-plan.md index 5ab819b..45b532e 100644 --- a/drafts/9p-backend-plan.md +++ b/drafts/9p-backend-plan.md @@ -1,297 +1,140 @@ -# Plan: a 9p backend (`bin/eval-under-9p`) +# 9p backends: design record -Status: **plan, not yet implemented**. This documents the design and the -sequencing for adding 9p as a matrix row, so the implementation PRs can -be reviewed against something. Move the durable parts (settings tables, -known-red reasoning) into GOTCHAS.md as they land; delete this file when -the last phase lands or is explicitly dropped. +Status: **implemented** (both backends, CI rows `9p-tcp / n/a` and +`9p-virtio / mapped`). This file is the decision record from the design +review; the operational truth lives where it belongs -- settings and +semantics in [GOTCHAS.md](../GOTCHAS.md), knobs in each script's +`--help`, deferred rows in GOTCHAS "Not yet covered". Delete this file +once those deferred rows have either landed or been rejected. ## Why 9p -9p is the filesystem people get, usually without choosing it, whenever a -directory is shared *into* a VM or container boundary: - -- **Vagrant + QEMU/libvirt**: `synced_folder ..., type: "9p"` in - vagrant-libvirt is QEMU's virtio-9p device (`-virtfs local,...`) on - the host side and the kernel's `v9fs` client (`mount -t 9p -o - trans=virtio`) on the guest side. This is the concrete case that - motivates the backend: our own Vagrantfile deliberately avoids 9p for - syncing ("rsync is more portable than 9p/virtiofs and avoids - permission surprises") -- those permission surprises are precisely - what this harness exists to measure rather than avoid. -- **WSL2**: `/mnt/c` and friends are 9p (Microsoft's own server). -- **Chrome OS crostini**, **kata-containers** (pre-virtiofs), various - lightweight-VM dev environments. - -The quirk classes are distinct from anything the current rows cover: -cache-mode staleness (`cache=loose` shows other-writer changes late; the -default no-cache mode makes some mmap patterns impossible), byte-range / -BSD locking that historically returns `ENOLCK` or is faked -server-side, no inotify propagation, ownership semantics that depend on -the *server's* security model rather than the client mount, `msize` -throughput cliffs, no `O_TMPFILE`. git-annex leans on locking, mmap -(via git), and HOME-relative sockets -- a 9p row should light up in -informative ways, per layer, exactly like the vfat row does. - -## The shape of the problem - -Every existing backend mounts on the host and runs the suite on the -host. The *interesting* 9p deployment splits across a VM boundary: the -server is the QEMU process on the host, the client is `v9fs` in the -guest kernel, and the transport is virtio. Testing "9p as vagrant users -experience it" therefore means running the suite **inside a guest**. -There are two mechanisms worth having, and they share one backend -script: - -### Transport `virtio` (primary -- the real vagrant/QEMU stack) - -Naively this inverts the harness: targets are installed on the runner -(`install-target.sh` builds into `$EVAL_UNDER_SRC_DIR`), but the suite -would have to run in a guest that has none of that. **virtme-ng** -dissolves the inversion: `vng` boots a QEMU guest whose root filesystem -is a copy-on-write view of the *host's own* root (virtiofs by default, -`--force-9p` as fallback), so runner-installed targets, the git-annex -daily build, `~/.gitconfig`, and the uid/gid layout all exist in the -guest unchanged. The backend then adds its own device for the -filesystem under test: - -- host side: a fresh scratch dir exported via - `-virtfs local,path=$ORIG,mount_tag=eval9p,security_model=,id=eval9p` - (passed through `vng --qemu-opts`); -- guest side: `mount -t 9p -o trans=virtio,version=9p2000.L[,msize=..][,cache=..] - eval9p $MNT`, then run the wrapped command with `TMPDIR` / - `DATALAD_TESTS_TEMP_DIR` / (`HOME` with `--set-home`) pointing at it. - -stdout/stderr stream to the job log as usual and `vng` propagates the -wrapped command's exit status (verify this early -- it is -load-bearing for CI redness). - -So the backend contract ("run CMD with TMPDIR on the mount") survives -intact; the only novelty is that CMD executes under a different kernel -instance. Writes to host paths *outside* the shared dirs land in the -CoW layer and evaporate -- which is a feature (free cleanup), except -for the git target's `t/test-results/**` that the workflow uploads: -pass `--rwdir "$EVAL_UNDER_SRC_DIR"` so those writes reach the host. - -### Transport `tcp` (secondary -- no VM, mirrors `eval-under-nfs`) - -`diod` (LLNL's 9P2000.L server; Ubuntu universe: 1.0.24-5 on jammy, -1.0.24-5.1 on noble) exports a fresh scratch dir on `127.0.0.1`, and -the host kernel mounts it: - - diod --foreground --no-auth --listen 127.0.0.1:5640 --export "$ORIG" & - mount -t 9p -o trans=tcp,port=5640,aname=$ORIG,version=9p2000.L,uname=root,access=user \ - 127.0.0.1 "$MNT" - -Same architecture as the NFS backend (localhost server + kernel client), -so it drops into the harness with zero conceptual novelty. It exercises -the same `v9fs` client code but a *different server* than QEMU's virtfs --- different bug surface, cheaper row. It requires the 9p client -modules in the **host** kernel (see risks: azure kernels). - -**Recommendation:** implement `virtio` as the deliverable -- it is the -stack the backend exists to represent -- and `tcp` opportunistically; -the script skeleton (scratch dir, teardown trap, env plumbing, option -parsing) is shared, only `start_*`/`mount_*` differ per transport. - -## Backend script: `bin/eval-under-9p` - -House pattern (`set -eu`, `SUDO` arrays, `trap teardown EXIT`, here-doc -`usage()`, common flags on top). Backend-specific knobs, following the -"distro/kernel defaults on purpose, knobs to deviate" philosophy the -loop backend established: - -| Flag | Env var | Default | Purpose | -| --- | --- | --- | --- | -| `--transport {virtio,tcp}` | `EVAL_UNDER_9P_TRANSPORT` | `virtio` | Which of the two stacks above. | -| `--security-model M` | `EVAL_UNDER_9P_SECURITY_MODEL` | `mapped-xattr` | virtio only. QEMU virtfs server model: `mapped-xattr` (ownership/mode faked in xattrs; what vagrant-libvirt `accessmode: "mapped"` gives), `passthrough` (real uids; QEMU must run as root), `none`. | -| `--cache MODE` | `EVAL_UNDER_9P_CACHE` | unset (kernel default) | v9fs client cache mode. Unset = whatever the guest kernel defaults to; a knob because `loose` vs default is the single biggest semantic axis users hit. | -| `--msize BYTES` | `EVAL_UNDER_9P_MSIZE` | unset (kernel default; 128 KiB since 5.15) | Client request size; a throughput knob, occasionally a correctness one. | -| `--kernel VER` | `EVAL_UNDER_9P_KERNEL` | pinned in `.github/matrix.yaml` | virtio only. Guest kernel for `vng --run`. See "pin the guest kernel" below. | -| `--memory MB` / `--cpus N` | `EVAL_UNDER_9P_{MEMORY,CPUS}` | 8192 / nproc | virtio only. Guest sizing; public-repo runners have 4 vCPU / 16 GB. | -| `--port P` | `EVAL_UNDER_9P_PORT` | 5640 | tcp only. Non-privileged, non-564 so nothing collides. | - -Mount `version=9p2000.L` is fixed, not a knob: `.u` is legacy and diod -speaks only `.L`. - -Semantics to preserve from the existing backends: - -- **User identity.** Under `mapped-xattr` the server fabricates - ownership, so run the wrapped command as the invoking user (NFS-style - drop) -- that is what a vagrant user's synced folder looks like. - `needs-root` targets (pjdfstest, stress-ng) run as root *in the - guest*; under `mapped-xattr` their chown/mknod get absorbed into - xattr mapping -- measuring that is the point, and GOTCHAS must say so - before anyone reads those cells as kernel bugs. Under `passthrough`, - QEMU itself runs as root (we already have `sudo -E` in the workflow). -- **`--keep`.** tcp: leave diod + mount up, as NFS does. virtio: the - guest is gone when vng exits; keep the *host-side backing dir* - (readable directly; under `mapped-xattr` the real metadata sits in - `user.virtfs.*` xattrs) and echo the full `vng` command line so the - session can be relaunched interactively for poking. -- **Failure diagnostics.** The v9fs client logs to the *guest* dmesg, - which dies with the VM. Wrap the guest-side command so that on - non-zero exit it appends `dmesg | tail -50` to a host-visible file - (under the `--rwdir` or the 9p mount's backing dir), and teach - `bin/ci/dump-failure-logs.sh` to print it. Same lesson as - `wait_for_mount_usable()`: make the next red cell diagnosable from - the job log alone. -- **Mount-usability probe.** Reuse the BeeGFS create+write+read-back - probe inside the guest right after the mount, before handing over to - the suite. 9p mounts fail late and weird; a probe converts that into - an early loud error. - -Guest-side execution sketch (virtio), all inside one `vng` invocation -so there is exactly one boot per cell: - - vng --run "$KERNEL" --cpus "$CPUS" --memory "$MEM" \ - --rwdir "$EVAL_UNDER_SRC_DIR" \ - --qemu-opts "-virtfs local,path=$ORIG,mount_tag=eval9p,security_model=$SECMODEL,id=eval9p" \ - -- bin/eval-under-9p --guest-stage2 ... - -with `--guest-stage2` (hidden flag) doing: modprobe 9p/9pnet_virtio if -modular, mount, probe, mkdir RUN_HOME, exec the command with the env -trio, capture dmesg on failure. Re-entering the same script keeps the -host/guest halves in one reviewable file. - -## Matrix integration - -`.github/matrix.yaml` rows -- the `version` slot becomes the variant -token (slug-safe: no slashes, per the `cell_slug()` lesson): - - - backend: 9p - version: virtio-mapped - label: "9p virtio (mapped)" - - backend: 9p - version: virtio-passthrough - label: "9p virtio (passthrough)" - -Start by landing `virtio-mapped` only (4 new cells); add -`virtio-passthrough` once the first row's failure modes are understood, -and `tcp-diod` if/when the host-module probe says the runners can do it. -`run-under.sh` grows a `9p)` case that splits the version token into -`--transport` / `--security-model` flags, exactly parallel to the -`loop)` case translating `version` into `--fs`. - -**Pin the guest kernel.** v9fs client behavior moves significantly -between kernel versions (the 6.6-6.8 cache rework renamed and -re-defaulted the cache modes). An unpinned guest kernel makes a -newly-red cell ambiguous in exactly the way the pinned `refs:` exist to -prevent -- so add e.g. `refs: { 9p-kernel: "6.8" }` and have the -backend default `--kernel` from it (`vng --run ` fetches a -prebuilt kernel; the host's running kernel remains an explicit opt-in -via `--kernel host`). Bump deliberately, and expect GOTCHAS entries to -be keyed to it. - -**Per-row `runs-on`.** The `test` job is currently hard-coded to -ubuntu-22.04 for BeeGFS-DKMS reasons that do not bind the 9p rows, and -ubuntu-24.04 is a materially better host here: `virtme-ng` (1.22-1) and -rust `virtiofsd` are packaged, QEMU is 8.2, and the stock kernel is -newer. Add an optional `runs-on:` key per backend row (default -ubuntu-22.04), emit it from `matrix-json.sh`, and set -`runs-on: ${{ matrix.runs-on }}` in the workflow. Contained change, -keeps the BeeGFS rows untouched, and removes the need to pip-install -virtme-ng on jammy. - -`bin/ci/install-backend.sh` gains `install_9p()`: - -- `apt_install qemu-system-x86 qemu-utils virtme-ng` (24.04; on 22.04 - fall back to `pipx install virtme-ng`), plus `diod` when the tcp - variant lands; -- best-effort `apt_install linux-modules-extra-$(uname -r)` -- - required for tcp host mounts and for `--kernel host` guests; known - to transiently fail when the archive lags the runner image - (actions/runner-images#8080), so don't hard-fail the virtio path on - it; -- KVM enablement: the standard udev rule - (`KERNEL=="kvm", GROUP="kvm", MODE="0666"` + udevadm reload/trigger). - Since `run-under.sh` runs under `sudo -E` this is belt-and-braces; - still verify `/dev/kvm` exists and warn loudly when falling back to - TCG (a TCG git-annex run will blow the 2400 s budget -- treat TCG as - boot-smoke only, and let the cell fail fast with a clear message - rather than time out mutely). - -Timeouts: reuse the per-target values initially; virtio adds ~10-20 s -of boot, and 9p latency sits between ext4 and sync-NFS. Adjust from -evidence, not in advance. - -Rest of the standard checklist from "Adding a new backend" in the -README: `gen-readme-matrix.sh` regeneration, GOTCHAS "Backend settings" -section (mount options, security model, msize/cache defaults *as -measured*, guest kernel), README file-layout row, and -`shellcheck bin/ci/*.sh bin/eval-under*`. New files need no SPDX -headers (`bin/**`, `drafts/**`, `provision/**` are covered by -REUSE.toml's aggregate block). - -## Vagrant / local iteration - -- `provision/setup.sh`: add `qemu-system-x86 qemu-utils virtme-ng diod` - and `linux-modules-extra-$(uname -r)` (the cloud image's `-virtual` - kernel keeps 9p client modules there). Nested KVM already works: the - Vagrantfile sets `lv.nested = true` + `cpu_mode = "host-passthrough"`, - so `vng` inside the VM is hardware-accelerated. -- Vagrantfile, opt-in cross-check share: behind an env guard (say - `VAGRANT_9P_SHARE=1`), add a *second* synced folder of - `type: "9p"` at `/vagrant-9p` -- the genuine vagrant-libvirt article, - for validating that `eval-under-9p --transport virtio` reproduces the - semantics of the real thing (compare a pjdfstest run on both). The - default stays rsync; the existing comment explaining why remains - true for the *repo* share. Confirm vagrant-libvirt's current - `accessmode` default and owner/group options at implementation time - (docs were unreachable from the drafting environment). - -Local usage after landing: - - sudo bin/eval-under 9p --set-home -- bash -c 'cd "$HOME" && git annex test' - sudo bin/eval-under 9p --transport tcp --set-home -- git annex test - sudo bin/eval-under 9p --cache loose -- ... # the classic vagrant foot-gun - sudo -E bin/ci/run-under.sh 9p virtio-mapped pjdfstest - -## Phase 0: a probe, before any backend code - -One `workflow_dispatch` job (script in `bin/ci/`, per house rules -- -e.g. `bin/ci/probe-9p.sh`, kept afterwards as a doctor script), run on -both ubuntu-22.04 and ubuntu-24.04, reporting: - -1. `/dev/kvm` presence and usability (as root and as the runner user - with the udev rule); -2. whether `linux-modules-extra-$(uname -r)` installs, and whether - `modprobe 9p 9pnet 9pnet_tcp 9pnet_virtio` then succeeds on the - azure kernel (decides the tcp row's CI fate; the answer is genuinely - unknown -- packages.ubuntu.com contents search draws a blank); -3. `vng --run -- uname -a` boot smoke + a 5-line 9p - mount-and-touch inside the guest, and confirmation that a non-zero - guest exit propagates to the host. - -This converts every open risk below into a fact for the cost of one CI -run, before the backend script exists. - -## Risks and open questions - -| Risk | Exposure | Mitigation | -| --- | --- | --- | -| Azure kernel lacks 9p client modules | tcp row on hosted runners only | Probe decides; virtio row is immune (pinned `vng` kernel ships its own modules); tcp stays available locally/VM regardless. | -| `modules-extra` transiently uninstallable (runner image vs archive lag) | tcp row, `--kernel host` | Best-effort install; virtio row does not depend on it. | -| KVM on standard runners is unofficial | whole virtio row | Works today (udev rule; root via `sudo` regardless); probe verifies per-image; TCG fallback = fail fast with a clear message. | -| `vng` exit-status / stdout plumbing quirks | CI signal integrity | Verified explicitly in Phase 0 item 3. | -| virtiofsd availability for the vng *root* on jammy | only if 9p rows stay on ubuntu-22.04 | Prefer per-row `runs-on: ubuntu-24.04`; `vng --force-9p` for the rootfs is the fallback (slower, and amusingly turns even `/usr` into 9p). | -| Suite behavior differs guest-vs-host for non-fs reasons (loopback services, sockets) | git-annex target mostly | `git annex test` is local-only; add `--net user` to vng only if a target proves to need it. | - -## Sequencing - -1. **PR 1 -- probe.** `bin/ci/probe-9p.sh` + a tiny dispatch workflow; - record findings in the PR, then wire the answers into this plan. -2. **PR 2 -- the backend.** `bin/eval-under-9p` (virtio transport, - `mapped-xattr`), `9p)` cases in `run-under.sh` + - `install-backend.sh`, per-row `runs-on`, `9p-kernel` ref, - matrix row `virtio-mapped`, GOTCHAS settings section, README regen, - provision additions, `dump-failure-logs.sh` 9p case. -3. **PR 3 -- variants.** `virtio-passthrough` row; `tcp-diod` row if - the probe cleared it; Vagrantfile opt-in 9p share for - cross-validation. -4. **Later, own decisions:** `cache=loose` and msize variant rows - (GOTCHAS "Not yet covered" until then), and a sibling - `eval-under-virtiofs` backend -- the designated successor to 9p in - the same vagrant/QEMU role, nearly free once the vng plumbing - exists, and the natural control row for "is this 9p, or is this - any-VM-shared-fs?". +9p is the filesystem people get, usually without choosing it, when a +directory is shared across a VM boundary: Vagrant + QEMU/libvirt +`type: "9p"` synced folders (QEMU's virtfs server + the kernel v9fs +client over virtio), WSL2's `/mnt/c` (Microsoft's 9p server), Chrome OS +crostini, pre-virtiofs kata. Its quirk classes -- cache staleness, +server-faked or whole-file locking, absorbed chown/mknod under mapped +security models, msize cliffs, no `O_TMPFILE`, no remote change +notification -- are disjoint from every existing row, and our own +Vagrantfile dodges them ("rsync ... avoids permission surprises") +rather than measuring them. + +## The two mechanisms, and why both exist + +- **`bin/eval-under-9p-tcp`** -- diod (9P2000.L) on localhost TCP, + kernel client mount on the same host. Architecturally the NFS + backend's sibling; no VM, no KVM, server runs unprivileged. The + everyday local-debugging 9p, and the cheap CI row. +- **`bin/eval-under-9p-virtio`** -- QEMU `-virtfs local,...` into a + virtme-ng guest booted from the host's own rootfs, suite runs inside + the guest. The only way to exercise the *actual* server vagrant users + hit; virtme-ng is what makes it fit the harness (host-installed + targets exist in the guest unchanged, exit status and stdio are + plumbed out over virtio-serial). + +Two scripts rather than one `--transport` flag: they share the string +"9p" and the env-trio contract and nothing else -- different daemon vs +hypervisor, privilege model, teardown, diagnostics (host dmesg vs a +guest that no longer exists). One mechanism per script is the house +pattern (loop parametrizes filesystems, beegfs parametrizes versions; +neither multiplexes mechanisms). + +## Decisions of record + +Two independent design reviews (one systems-mechanics, one +harness-fit) converged on the shape that got implemented. The calls, +and what settled them: + +1. **Both transports landed together, tcp as the low-risk row.** The + interesting client semantics are identical across both (same v9fs); + the servers differ instructively (diod: whole-file `flock`; QEMU: + TLOCK-always-succeeds). The virtio row carries the CI unknowns, so + the tcp row guarantees the matrix gains a working 9p row even if + virtio needs iteration. +2. **Backends never read `.github/matrix.yaml`.** That line is what + keeps `bin/eval-under-*` first-class local tools. The pinned guest + kernel (`refs: 9p-kernel`, mainline build fetched by `vng --run`) + is handed to the backend *by `bin/ci/run-under.sh`*, exactly like + `--size` and `--no-root-squash`; the script's own default is the + host kernel -- zero downloads, works offline. +3. **Guest kernel pinned in CI, host kernel locally.** The v9fs client + *is* the kernel and moved substantially in 6.4 (cache-mode rework) + and 6.8 (netfs buffered writes); unpinned, a newly-red cell cannot + distinguish filesystem regression from client drift. This is a + deliberate departure from the "distro defaults, not pinned" + environment philosophy (NFS protocol version is deliberately + unpinned) -- argued, not smuggled: for 9p the client is the thing + under test's other half. +4. **`--disable-microvm`.** vng's microvm machine has no PCI bus the + stock Ubuntu/mainline guest kernels can enumerate, and `-virtfs` is + virtio-9p-pci -- on the exact recommended path (24.04 + KVM + + virtiofsd) the mount tag would silently never appear. +5. **Root only where root is needed.** tcp: diod runs unprivileged in + single-user mode, sudo is for mount/umount; virtio: nothing on the + host needs root at all (mounting happens inside the guest, QEMU and + vng run unprivileged). Guest scripts run as root, so the wrapped + command is *dropped* to the invoking user via `runuser` -- vng's + default would otherwise measure root-in-a-VM. `--run-as-root` (both + scripts) is the 9p analog of NFS `--no-root-squash`, wired to + `target_needs_root()` in run-under.sh; without it the pjdfstest + cell would be a contentless setup-failure red. +6. **Guest mountpoint defaults to `/tmp/eval-under-9p`.** vng's rootfs + is read-only outside its overlay set; `/mnt` would EROFS. `--share-rw` + exists because overlay writes evaporate with the guest -- CI shares + `$EVAL_UNDER_SRC_DIR` so git's `t/test-results/**` reaches the + runner for artifact upload. +7. **Host-side `--vm-timeout` around the whole guest.** The per-target + `timeout` that run-under.sh wraps around the suite travels *into* + the guest and cannot catch a boot or mount hang; CI passes + target-timeout + 300s. +8. **Diagnostics survive the guest.** All guest output is teed to + `/var/log/eval-under-9p-virtio.log` (in the artifact list), stage2 + appends the guest dmesg tail on failure, and the backend + disambiguates vng's exit sentinels (124 = VM killed by + `--vm-timeout`; 255 = crash-or-genuine-255). The backing dir is + teardown-deleted before `dump-failure-logs.sh` runs, so nothing + diagnostic may live only there. +9. **Per-row `runs-on` in matrix.yaml** (default ubuntu-22.04; 9p rows + ubuntu-24.04, where virtme-ng/virtiofsd are packaged). Costed + knowingly: the backend tuple grew a fourth field across matrix.sh / + matrix-json.sh / gen-readme-matrix.sh, and rows on different images + differ in host kernel and tool versions -- acceptable here because + the virtio row's client kernel is pinned anyway, recorded so the + next person prices it too. +10. **No separate probe workflow.** The durable assertions live in + `install-backend.sh` (modprobe + /proc/filesystems for tcp; udev + KVM rule, `vng --version`, and a pre-warm boot of the pinned + kernel for virtio -- which doubles as the boot smoke and moves the + ~180 MB mainline download outside the suite's timeout). The PR's + own matrix run is the integration probe. + +## Facts the reviews settled (so nobody re-derives them) + +- 9p client modules (`9p`, `9pnet`, `9pnet_fd`, `9pnet_virtio`) ship in + the kernel's **base `linux-modules`** package on Ubuntu generic, + virtual and azure flavours alike -- `linux-modules-extra` is NOT + needed anywhere in this design. (`9pnet_tcp` does not exist; + `trans=tcp` lives in `9pnet_fd`.) +- Standard GitHub-hosted Linux runners have KVM (officially since + 2024); the udev rule in `install_9p_virtio()` is the documented + enablement for non-root use. +- `vng --run` wants a tag-shaped version (`v6.8`), downloads the Ubuntu + mainline image+modules (~180 MB, cached in `~/.cache/virtme-ng`), + and those builds carry the 9p modules; vng propagates the guest + command's exit code over a dedicated channel, with 255 as its + crash sentinel. +- vagrant-libvirt's *default* accessmode is `passthrough` under an + unprivileged QEMU -- neither of the first two virtio rows; recorded + with a local repro recipe in GOTCHAS "Not yet covered". +- diod caps msize at 64 KiB and implements Tlock as whole-file + `flock()`; QEMU's virtfs answers every TLOCK with success. Same + client, two instructively different servers. + +## Deferred (tracked in GOTCHAS "Not yet covered") + +`9p-virtio / passthrough` (root QEMU) and the true vagrant default +(passthrough, unprivileged QEMU); `cache=loose` and small-msize +variants; an `eval-under-virtiofs` sibling backend as the "is this 9p +or any VM-shared fs?" control row. All are flag-reachable locally +today; each becomes one matrix line when promoted. diff --git a/provision/setup.sh b/provision/setup.sh index d6f7ab6..9f06d9e 100755 --- a/provision/setup.sh +++ b/provision/setup.sh @@ -43,6 +43,18 @@ apt-get install -y --no-install-recommends \ nfs-kernel-server \ dosfstools xfsprogs btrfs-progs +log "eval-under-9p-tcp / eval-under-9p-virtio dependencies" +# diod: 9P2000.L server for eval-under-9p-tcp. +# virtme-ng: boots the eval-under-9p-virtio guest (vng); the VM's +# nested KVM (Vagrantfile: lv.nested) accelerates it. +# qemu-system-x86: virtme-ng only Recommends it; --no-install-recommends. +# virtiofsd: vng rootfs transport (falls back to 9p without it). +# busybox-static: vng initramfs for pinned --kernel downloads. +# zstd: mainline kernel debs ship .ko.zst modules. +# 9p client modules ship in the kernel's base linux-modules package. +apt-get install -y --no-install-recommends \ + diod virtme-ng qemu-system-x86 qemu-utils virtiofsd busybox-static zstd + log "eval-under test-target dependencies" # stress-ng: bin/ci/target-stress-ng.sh (apt is the whole install). # autoconf/automake/libtool: building pjdfstest from its pinned tag. From 781de66cbef2c81a620241aae80e9674d33190db Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 19:58:23 +0000 Subject: [PATCH 3/6] 9p-tcp: don't chown the diod log; runners' protected_regular blocks it On the hosted runners (fs.protected_regular=2) the sequence root-mktemp -> chown-to-invoker -> root '>>' append in sticky /tmp is denied, so diod never started and every 9p-tcp cell failed with "diod did not bind". The chown was never needed: the current shell opens the log fd and diod only inherits it, whatever uid diod runs as. Reproduced and verified both ways by flipping fs.protected_regular locally. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- bin/eval-under-9p-tcp | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/bin/eval-under-9p-tcp b/bin/eval-under-9p-tcp index 1a0a334..49dae48 100755 --- a/bin/eval-under-9p-tcp +++ b/bin/eval-under-9p-tcp @@ -163,8 +163,14 @@ start_diod() { log "creating backing dir $ORIG (owned by uid=$INVOKER_UID gid=$INVOKER_GID)" "${SUDO[@]}" mkdir -p "$ORIG" "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$ORIG" - "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$DIOD_LOG" + # The log stays owned by whoever this script runs as: the redirects + # below are performed by the current shell and diod only inherits the + # open fd, so diod's own uid never needs write access to the file. + # (Do NOT chown it to the invoker: with the script running as root, + # fs.protected_regular then blocks root's O_CREAT append in sticky + # /tmp -- "Permission denied" before diod ever starts.) + # # --foreground so it stays our child (we background + kill it # ourselves); --no-auth because munge has no place in a loopback # throwaway export. @@ -175,8 +181,8 @@ start_diod() { elif [ "$INVOKER_UID" != "$(id -u)" ]; then log "starting diod as $INVOKER_USER (single-user): 127.0.0.1:$PORT <- $ORIG" # The redirect deliberately happens in the current (root) shell, not - # under sudo -- the log was chown'd to the invoker above, and both - # sides can write it. + # under sudo: the shell owns the log and opens the fd, diod merely + # inherits it. # shellcheck disable=SC2024 sudo -u "$INVOKER_USER" diod --foreground --no-auth \ --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & From a690142968578ea4b137aa2f0733c5f26b8b7e06 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 20:07:31 +0000 Subject: [PATCH 4/6] 9p-virtio: replace --share-rw of the src tree with a --copy-out channel The 9p-virtio/git cell died in check-chainlint: vng's --rwdir shares are not uid-faithful (measured on the runner stack and reproduced locally) -- writes arrive host-side as the sharing daemon's identity, so a non-root guest process gets EACCES inside a directory it just created. Let the suite's in-tree writes (chainlinttmp, test-results) go to the guest's uid-faithful tmpfs overlay instead, and add --copy-out: stage2 ferries the requested guest paths back through the 9p export after the command (failed runs included), and the host half restores them to their real paths with ownership normalized to the invoker, so the artifact upload can read them. run-under.sh now passes --copy-out for the git target's t/test-results; --share-rw stays as a power tool with the caveat documented in its usage and in GOTCHAS. Verified end to end locally: a dropped-user guest write into the overlay lands back on the host with content intact. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- GOTCHAS.md | 11 +++++++ bin/ci/run-under.sh | 11 ++++--- bin/eval-under-9p-virtio | 63 ++++++++++++++++++++++++++++++++++++--- drafts/9p-backend-plan.md | 10 ++++--- 4 files changed, 83 insertions(+), 12 deletions(-) diff --git a/GOTCHAS.md b/GOTCHAS.md index 6072f6d..d470ad6 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -150,6 +150,17 @@ the protocol at all (inotify for the guest's own operations works normally), which is also why `cache=loose` can serve stale data indefinitely. +**vng's `--rwdir` shares are not uid-faithful** (measured on the +runner's exact stack): writes arrive on the host as the sharing +daemon's identity -- root when the backend runs under sudo -- and a +non-root guest process gets `EACCES` inside a directory it just +"successfully" created. This is why the git target's +`t/test-results/**` travels via the backend's `--copy-out` (through +the 9p export itself, ownership normalized to the invoker) rather than +via `--share-rw`, and why the source tree's in-suite writes +(`chainlinttmp`, `test-results`) are left to the guest's uid-faithful +tmpfs overlay instead of a share. + **The guest is disposable; the log is not.** Everything the suite and the guest console print is teed to `/var/log/eval-under-9p-virtio.log` on the host (stage2 appends the guest `dmesg` tail on failure), because diff --git a/bin/ci/run-under.sh b/bin/ci/run-under.sh index 5fc5659..6dbbeb7 100755 --- a/bin/ci/run-under.sh +++ b/bin/ci/run-under.sh @@ -84,10 +84,13 @@ case "$BACKEND" in --kernel "$EVAL_UNDER_9P_KERNEL_REF" --memory 4G --vm-timeout $((TIMEOUT + 300))) - # Suites that write results onto the runner's disk (git's - # t/test-results) need that dir shared read-write into the - # guest; everything else in the guest's overlay evaporates. - [ -d "$EVAL_UNDER_SRC_DIR" ] && opts+=(--share-rw "$EVAL_UNDER_SRC_DIR") + # git writes its per-script logs into the source tree; + # inside the guest that tree is a discardable overlay (vng + # --rwdir is deliberately NOT used: it is not uid-faithful, + # see the backend's --share-rw caveat), so ferry the + # results back through the 9p export instead for the + # artifact upload. + [ "$TARGET" = git ] && opts+=(--copy-out "$EVAL_UNDER_SRC_DIR/git/t/test-results") target_needs_root "$TARGET" && opts+=(--run-as-root) ;; *) echo "unknown backend: $BACKEND" >&2; exit 1 ;; esac diff --git a/bin/eval-under-9p-virtio b/bin/eval-under-9p-virtio index bf3e67b..425d0f0 100755 --- a/bin/eval-under-9p-virtio +++ b/bin/eval-under-9p-virtio @@ -60,8 +60,9 @@ DEBUG_BOOT="${EVAL_UNDER_9P_VIRTIO_DEBUG_BOOT:-0}" SHELL_MODE=0 TAG="eval9p" -# Space-separated initial value from the environment; --share-rw appends. +# Space-separated initial values from the environment; the flags append. read -r -a SHARE_RW <<< "${EVAL_UNDER_9P_VIRTIO_SHARE_RW:-}" +read -r -a COPY_OUT <<< "${EVAL_UNDER_9P_VIRTIO_COPY_OUT:-}" usage() { cat <<'EOF' @@ -155,9 +156,20 @@ Options (flag / env var / default / purpose): --share-rw PATH EVAL_UNDER_9P_VIRTIO_SHARE_RW (none) Host directory to share read-write into the guest at the same - path (vng --rwdir). Repeatable; env var is space-separated. CI - passes the target-suite source dir so test-results/ written by - the suite reach the host for artifact upload. + path (vng --rwdir). Repeatable; env var is space-separated. + CAVEAT (measured): the share is not uid-faithful -- writes + arrive on the host as the daemon's identity (root when this + script runs under sudo), and a non-root guest process can fail + with EACCES inside a directory it just created. Fine for + root-only workflows; for suite outputs use --copy-out instead. + + --copy-out PATH EVAL_UNDER_9P_VIRTIO_COPY_OUT (none) + Guest path (typically under one of vng's overlays, whose writes + are otherwise discarded with the guest) to copy back to the same + path on the host after the command finishes. Travels through the + 9p export itself, so it needs no extra share; ownership on the + host is normalized to the invoking user. Repeatable; env var is + space-separated. CI uses it for git's t/test-results. --vm-timeout S EVAL_UNDER_9P_VIRTIO_VM_TIMEOUT 0 (off) Host-side timeout around the whole guest. The per-target timeout @@ -215,6 +227,7 @@ log() { printf '\nI: %s\n' "$*"; } if [ "${1:-}" = "--guest-stage2" ]; then shift G_TAG="" G_MNT="" G_MOPTS="" G_PATH="$PATH" G_DROP_TO="" G_SET_HOME=0 G_SHELL=0 + G_COPY_OUT=() while [ $# -gt 0 ]; do case "$1" in --tag) G_TAG="$2"; shift 2 ;; @@ -224,6 +237,7 @@ if [ "${1:-}" = "--guest-stage2" ]; then --drop-to) G_DROP_TO="$2"; shift 2 ;; --set-home) G_SET_HOME=1; shift ;; --shell) G_SHELL=1; shift ;; + --copy-out) G_COPY_OUT+=("$2"); shift 2 ;; --) shift; break ;; *) echo "guest-stage2: unknown arg: $1" >&2; exit 2 ;; esac @@ -297,6 +311,22 @@ if [ "${1:-}" = "--guest-stage2" ]; then env HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" \ "$@" || rc=$? fi + # Ferry requested guest paths (typically overlay content that would + # otherwise vanish with the guest) out through the 9p export, as + # guest root, whatever the command's outcome -- failed runs are when + # the results matter most. + for p in ${G_COPY_OUT[@]+"${G_COPY_OUT[@]}"}; do + if [ -e "$p" ]; then + dest="$G_MNT/.eval-under-copyout$(dirname "$p")" + if mkdir -p "$dest" && cp -a "$p" "$dest/"; then + echo "I: copied out $p" + else + echo "W: copy-out of $p failed" >&2 + fi + else + echo "I: copy-out path $p does not exist in the guest; skipping" + fi + done if [ "$rc" -ne 0 ]; then guest_dmesg_tail fi @@ -321,6 +351,7 @@ while [ $# -gt 0 ]; do --writeout) WRITEOUT="$2"; shift 2 ;; --run-as-root) RUN_AS_ROOT=1; shift ;; --share-rw) SHARE_RW+=("$2"); shift 2 ;; + --copy-out) COPY_OUT+=("$2"); shift 2 ;; --vm-timeout) VM_TIMEOUT="$2"; shift 2 ;; --allow-tcg) ALLOW_TCG=1; shift ;; --log) LOG="$2"; shift 2 ;; @@ -344,8 +375,10 @@ esac # sees the host's /etc/passwd through the shared rootfs). if [ -n "${SUDO_USER:-}" ]; then INVOKER_USER="$SUDO_USER"; INVOKER_UID="${SUDO_UID:-0}" + INVOKER_GID="${SUDO_GID:-$INVOKER_UID}" else INVOKER_USER="$(id -un)"; INVOKER_UID="$(id -u)" + INVOKER_GID="$(id -g)" fi [ "$INVOKER_UID" = 0 ] && RUN_AS_ROOT=1 # dropping root to root is a no-op @@ -481,6 +514,10 @@ build_stage2_string() { [ "$SET_HOME" = 1 ] && args+=(--set-home) [ "$RUN_AS_ROOT" = 0 ] && args+=(--drop-to "$INVOKER_USER") [ "$SHELL_MODE" = 1 ] && args+=(--shell) + local c + for c in ${COPY_OUT[@]+"${COPY_OUT[@]}"}; do + args+=(--copy-out "$c") + done args+=(--) local a STAGE2_STR="" @@ -535,6 +572,24 @@ else rc="${PIPESTATUS[0]}" fi +# Retrieve --copy-out payloads from the export's backing dir onto their +# real host paths, normalizing ownership: under mapped-xattr the +# backing files carry the QEMU process's uid and 0600/0700 modes, which +# a later artifact-upload step could not even read. +for p in ${COPY_OUT[@]+"${COPY_OUT[@]}"}; do + src="$ORIG/.eval-under-copyout$p" + if [ -e "$src" ]; then + mkdir -p "$(dirname "$p")" + cp -r --no-preserve=mode,ownership "$src" "$(dirname "$p")/" + if [ "$(id -u)" -eq 0 ]; then + chown -R "$INVOKER_UID:$INVOKER_GID" "$p" + fi + log "copied back $p" + else + echo "W: no copy-out payload for $p (guest never wrote it?)" >&2 + fi +done + if [ "$rc" -eq 124 ] && [ "$VM_TIMEOUT" != 0 ]; then echo "ERROR: guest exceeded --vm-timeout ${VM_TIMEOUT}s and was killed (boot or mount hang?)" >&2 elif [ "$rc" -eq 255 ]; then diff --git a/drafts/9p-backend-plan.md b/drafts/9p-backend-plan.md index 45b532e..683f3f6 100644 --- a/drafts/9p-backend-plan.md +++ b/drafts/9p-backend-plan.md @@ -80,10 +80,12 @@ and what settled them: `target_needs_root()` in run-under.sh; without it the pjdfstest cell would be a contentless setup-failure red. 6. **Guest mountpoint defaults to `/tmp/eval-under-9p`.** vng's rootfs - is read-only outside its overlay set; `/mnt` would EROFS. `--share-rw` - exists because overlay writes evaporate with the guest -- CI shares - `$EVAL_UNDER_SRC_DIR` so git's `t/test-results/**` reaches the - runner for artifact upload. + is read-only outside its overlay set; `/mnt` would EROFS. Overlay + writes evaporate with the guest, and vng's `--rwdir` shares turned + out not to be uid-faithful (measured; see GOTCHAS) -- so git's + `t/test-results/**` reaches the runner via the backend's + `--copy-out`, which ferries guest paths back through the 9p export + itself with ownership normalized. 7. **Host-side `--vm-timeout` around the whole guest.** The per-target `timeout` that run-under.sh wraps around the suite travels *into* the guest and cannot catch a boot or mount hang; CI passes From 23d5ea82ac356d2d53429ab3ea6349a34916adbb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 20:51:52 +0000 Subject: [PATCH 5/6] 9p-virtio: hedge the exit-124 message; in-guest and VM timeouts collide Both the wrapped command's own in-guest timeout and the host-side --vm-timeout surface as exit 124; the first CI git-annex hang showed the message wrongly asserting the VM was killed when the guest had in fact reported cleanly (its dmesg tail was right there above). Say both possibilities and how to tell them apart. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- bin/eval-under-9p-virtio | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/eval-under-9p-virtio b/bin/eval-under-9p-virtio index 425d0f0..e95439a 100755 --- a/bin/eval-under-9p-virtio +++ b/bin/eval-under-9p-virtio @@ -591,7 +591,9 @@ for p in ${COPY_OUT[@]+"${COPY_OUT[@]}"}; do done if [ "$rc" -eq 124 ] && [ "$VM_TIMEOUT" != 0 ]; then - echo "ERROR: guest exceeded --vm-timeout ${VM_TIMEOUT}s and was killed (boot or mount hang?)" >&2 + echo "W: exit 124 -- either the wrapped command hit its own timeout inside the" >&2 + echo " guest (a dmesg tail above means the guest was alive to report it), or" >&2 + echo " the VM exceeded --vm-timeout ${VM_TIMEOUT}s and was killed from outside." >&2 elif [ "$rc" -eq 255 ]; then echo "W: exit 255 -- either the wrapped command exited 255, or the guest died" >&2 echo " before reporting (vng's sentinel). Re-run with --debug-boot to see boot." >&2 From 5932f26d7d15612935181586ae55df7ea11d91b2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 20:53:33 +0000 Subject: [PATCH 6/6] GOTCHAS: record the 9p known-red cells from the first full matrix run All eight 9p cells ran their suites end-to-end on the PR's matrix run; every red is a finding, not a harness failure, and the legacy twenty cells match master's baseline exactly. Recorded per house rule: - git-annex test hangs identically on both servers (testremote init fails, then unavailable-remote/removeKey blocks until the timeout) -- the pre-registered locking suspect; - stress-ng: fstat/ftruncate ENOENT on open-but-unlinked files; fallocate fails only on diod (server divergence), with v6.8 guest kernel WARNs in the v9fs fid-lookup path on the QEMU row; - pjdfstest: long-pathname 03.t scripts, the open/06.t flags matrix, and unlink/14.t (shared with the NFS row) on diod; virtio tally in the artifact; - git testsuite: 194 (diod) / 177 (virtio) failed assertions, t1050-large diod-only, t1517/t0450 shared and green on ext4. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01MovQRK32XDaU4PRDrCP6S7 --- GOTCHAS.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/GOTCHAS.md b/GOTCHAS.md index d470ad6..74d0dcd 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -265,6 +265,61 @@ layers on top, or in the syscalls the pjdfstest column is flagging. Pre-dates the matrix; ext4 is the control row, so this one *is* a real bug worth chasing rather than a filesystem property. +### `9p * / git-annex test` -- hangs, on both servers identically + +The suite's very first group (`testremote type git`) fails `init` +within seconds ("git-annex: Not initialized"), then the run goes +silent inside `unavailable remote / removeKey` until the per-target +timeout kills it (exit 124) ~40 minutes later. Identical on diod and +on QEMU virtfs, so it is v9fs-client/protocol semantics rather than a +server quirk -- and diod's own docs warn that distributed record +locking "will deadlock", which is the pre-registered suspect. A hang +*is* the finding: git-annex on a real vagrant 9p share stalls the same +way. Not yet broken down further; the cell burns its full timeout by +design (a runaway suite reports as `timeout`, not a bare cancellation). + +### `9p * / stress-ng` -- the unlinked-open-file class, server-dependent + +Both rows run all 20 stressors to a clean tally; the failures are +`fstat`/`ftruncate` returning **ENOENT on files that are open but +unlinked** (stress-ng's create-unlink-keep-fd pattern): + +- diod: failed `fallocate`, `hdd`, `copy-file` (17 passed); +- QEMU virtfs: failed `copy-file`, `hdd` -- **`fallocate` passes**, a + clean server divergence worth keeping both rows for. + +On the QEMU row the guest kernel (pinned v6.8) also logs WARN traces +in `v9fs_fid_lookup_with_uid -> v9fs_vfs_getattr_dotl` while these +stressors run -- the client side of the same fid-on-unlinked-file gap, +and part of why the guest kernel is pinned: this signature is +kernel-version-specific. + +### `9p * / pjdfstest` + +Runs to completion on both rows (238 files, ~8800 assertions, ~3 min). +On diod the divergence concentrates in: every `*/03.t` (the +long-pathname scripts) across chown/ftruncate/link/mkdir/mkfifo/mknod/ +open/rmdir/symlink/truncate/unlink; `mkdir|mkfifo|mknod|open/00.t` +assertions 25-27; the `open/06.t` flags matrix (62 of 144); +`rename/10.t` (6 of 2099); `unlink/14.t` #4 (the same assertion the +NFS row flags); `utimensat/08.t`. The QEMU-virtfs row's tally is in +its `logs-*` artifact; per-assertion breakdown not yet done (same +status as the BeeGFS rows). + +### `9p * / git testsuite` + +The full `t0*.sh t1*.sh` selection runs under prove on both rows and +ends with totals; per-script `.out` files are in the artifacts (on the +virtio row they travel out of the guest via the backend's +`--copy-out`). Roll-up: diod 12 scripts / 194 failed assertions, QEMU +virtfs 10 / 177. Shared failures include `t1517-outside-repo` (104) +and `t0450-txt-doc-vs-help` (51) -- both green on the ext4 control +row, so 9p-related, not yet run down. **`t1050-large` (15 failures) +is diod-only** -- consistent with diod's 64 KiB msize cap and the +unlinked-file gap above; it passes under QEMU virtfs. Several of the +remaining entries are `# TODO known breakage` noise the dump's +`not ok` count includes. + ## Red that is not a finding Distinct from the cells above: these are harness races, and the fix is in