diff --git a/.github/matrix.yaml b/.github/matrix.yaml index 8efa5bb..681479a 100644 --- a/.github/matrix.yaml +++ b/.github/matrix.yaml @@ -34,6 +34,15 @@ src-dir: /opt/eval-under-src # did upstream add a test?). refs: git: v2.55.0 + # Guest kernel for the 9p-virtio backend's CI rows (vng downloads the + # Ubuntu mainline build; tag-shaped, leading "v" required). The v9fs + # *client* is the kernel, and its semantics move between releases + # (cache-mode rework in 6.4, netfs buffered writes in 6.8) -- an + # unpinned guest kernel would make a newly-red 9p cell ambiguous in + # exactly the way these pins exist to prevent. Local runs default to + # the host kernel instead; this pin is handed to the backend + # explicitly by bin/ci/run-under.sh. + 9p-kernel: v6.8 # pjd/pjdfstest carries exactly one tag upstream, "0.1" (2016), and it # no longer builds: major()/minor()/makedev() moved to # in glibc 2.28 and the tree compiles with -Werror, @@ -90,6 +99,12 @@ targets: # Row order of the README CI matrix. `version` is the literal "n/a" for # backends with nothing to pin (see bin/ci/install-backend.sh). +# +# `runs-on` (optional, default ubuntu-22.04) picks the runner image per +# row: BeeGFS DKMS constrains its rows to 22.04, while the 9p rows want +# 24.04, where virtme-ng/virtiofsd are packaged. Mixing images costs +# some cross-row comparability (different host kernel and tool +# versions) -- use it only when a backend actually needs it. backends: - backend: beegfs version: 7.4.6 @@ -106,3 +121,18 @@ backends: - backend: loop version: ext4 label: Loop ext4 + # v9fs client against diod (9P2000.L over TCP on localhost) -- the + # no-VM 9p row, architecturally the NFS backend's sibling. + - backend: 9p-tcp + version: n/a + label: 9p tcp (diod) + runs-on: ubuntu-24.04 + # v9fs client against QEMU's virtfs server over virtio, inside a + # virtme-ng guest -- the vagrant-libvirt "9p" synced-folder stack. + # "mapped" = security_model=mapped-xattr (vagrant accessmode + # "mapped"); vagrant's *default* (passthrough under an unprivileged + # QEMU) is a future row, see GOTCHAS.md "Not yet covered". + - backend: 9p-virtio + version: mapped + label: 9p virtio (mapped) + runs-on: ubuntu-24.04 diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index ad78515..b24c229 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -46,11 +46,12 @@ jobs: test: needs: matrix name: ${{ matrix.name }} - # ubuntu-22.04: kernel 5.15/6.5, within BeeGFS 7.4.x and 8.x DKMS - # support. ubuntu-24.04 hosted runners ship 6.17-azure which BeeGFS - # kernel modules cannot build against. NFS + loop backends don't - # need this, but sharing the runner OS keeps the matrix uniform. - runs-on: ubuntu-22.04 + # Per-cell runner image, from the backend row's optional `runs-on` + # in .github/matrix.yaml (default ubuntu-22.04: its kernel stays + # within BeeGFS DKMS support, while ubuntu-24.04 ships an azure + # kernel the BeeGFS modules cannot build against). The 9p rows + # override to ubuntu-24.04, where virtme-ng/virtiofsd are packaged. + runs-on: ${{ matrix.runs-on }} timeout-minutes: 60 strategy: fail-fast: false @@ -98,12 +99,13 @@ jobs: name: logs-${{ matrix.slug }} path: | /var/log/beegfs-* + /var/log/eval-under-9p-virtio.log /opt/eval-under-src/git/t/test-results/** if-no-files-found: ignore # One tiny artifact per cell, read back by the `badges` job. A # matrix job cannot contribute to a `needs.*.outputs` map, so this - # is how a fan-out reports 20 individual verdicts to a fan-in. + # is how a fan-out reports its per-cell verdicts to a fan-in. - name: Record cell result if: always() run: | diff --git a/GOTCHAS.md b/GOTCHAS.md index ff0883f..74d0dcd 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -86,6 +86,91 @@ a kernel module built against the runner's kernel. | Client conf | `fixtures/beegfs/beegfs-client.conf.template` | Auth disabled, all daemons on `127.0.0.1`, non-default ports (8004-8008) so nothing collides with the runner. | | `sysMountSanityCheckMS` | `0` **on v8 only** | BeeGFS v8 dropped the standalone `beegfs-helperd` binary; with no helperd the sanity check cannot complete and the mount would hang. v7 keeps the check. | +### 9p tcp (`bin/eval-under-9p-tcp`) + +A fresh directory exported on `127.0.0.1` by diod (LLNL's 9P2000.L +server, Ubuntu universe) and mounted with the kernel v9fs client -- +architecturally the NFS backend's sibling. Same client code as the +virtio row below, different server. + +| Knob | Value | Why | +| --- | --- | --- | +| Server | `diod --foreground --no-auth --listen 127.0.0.1:5640 --export ` | Loopback throwaway export; munge auth has no place here. | +| Server identity | the *invoking user* (single-user mode), unless `--run-as-root` | diod's documented "simplest and safest" mode; also means the server itself needs no privilege. | +| Mount (default) | `-t 9p -o trans=tcp,port=5640,version=9p2000.L,aname=,uname=,access=` | diod's documented single-user pairing. `access=` restricts the mount to that uid -- root included, which is why the usability probe runs as the invoker. | +| Mount (`--run-as-root`) | `...,uname=root,access=client`, diod as root | diod's documented multi-user (NFS-like) pairing; the 9p analog of `--no-root-squash`, applied by `target_needs_root()`. | +| `msize` | not set -- kernel requests its default (128 KiB since 5.15), **diod caps at 64 KiB** | The `/proc/mounts` line the backend echoes records the effective value; results are relative to it. | +| `cache` | not set -- kernel default, which is **no caching in every kernel era** | `--cache loose` exists to reproduce the classic stale-read foot-gun deliberately. | + +**diod locking is whole-file.** diod implements 9P `Tlock` as BSD +`flock()` on the host file: byte-range `fcntl` locks collapse to +whole-file locks, and diod's own docs say distributed record locking +will deadlock ("test your use case!"). A locking red on this row is a +diod-server finding first -- compare with the virtio row, whose server +fakes locks differently (below), before blaming the client. + +### 9p virtio (`bin/eval-under-9p-virtio`) + +QEMU's virtfs server (`-virtfs local,...`) exporting a fresh host dir +into a virtme-ng guest booted from the host's own root filesystem; the +suite runs *inside the guest* on a `mount -t 9p -o trans=virtio` mount. +This is the vagrant-libvirt `type: "9p"` synced-folder stack. + +| Knob | Value | Why | +| --- | --- | --- | +| Server | QEMU `-virtfs local,path=,mount_tag=eval9p,security_model=mapped-xattr` (the `mapped` row) | vagrant-libvirt `accessmode: "mapped"`. Ownership/mode/devices are faked in `user.virtfs.*` xattrs on the host files. | +| Guest kernel (CI) | pinned in `.github/matrix.yaml` `refs: 9p-kernel` (`vng --run`, Ubuntu mainline build) | The v9fs *client is the kernel*: cache-mode rework landed in 6.4, netfs buffered writes in 6.8. Unpinned, a newly-red cell can't distinguish "filesystem regressed" from "client changed". Local runs default to the host kernel instead. | +| Guest machine | `vng --disable-microvm` | vng's microvm has no PCI bus stock guest kernels can enumerate; `-virtfs` is virtio-9p-**pci**, so without this the mount tag silently never appears. | +| Mount | `-t 9p -o trans=virtio,version=9p2000.L` (msize/cache unset = kernel defaults, as above; virtio transport caps msize at 512000) | The backend echoes the guest's `/proc/mounts` line and `uname -r` -- that pair is what a result is relative to. | +| Command identity | dropped to the invoking user via `runuser` in the guest; `--run-as-root` keeps root (`target_needs_root()` rows) | A vagrant user's synced folder is an unprivileged view; guest scripts otherwise run as root in virtme. | +| `writeout` | not set (QEMU default) | vagrant-libvirt sets `wrpolicy="immediate"`; `--writeout immediate` reproduces that when wanted. | + +**mapped-xattr absorbs privileged operations.** Under this security +model `chown` "succeeds" into an xattr, and `mknod` creates a host +*regular file* wearing a device-node xattr -- so pjdfstest's and +stress-ng's privileged assertions pass or fail against the *mapping +layer*, not a kernel. That is the measurement (it is exactly the +"returns success while doing the wrong thing" class stress-ng exists to +catch), but read those cells with this table in hand. Corollaries: unix +sockets and FIFOs work unprivileged; host-side the backing files read +as mode 0600/0700 owned by the QEMU user (`--keep` shows exactly that). + +**QEMU virtfs locking is a polite lie.** QEMU's 9p server answers every +`TLOCK` with success (single-client assumption); the guest kernel takes +the local VFS lock first, so locking is coherent *within* the guest and +invisible to the host. Locks therefore mostly "work" on this row where +diod's whole-file behavior differs -- a divergence between the two 9p +rows on locking tests is expected, not noise. + +**Other v9fs client facts red cells trace back to** (both 9p rows): +writable `MAP_SHARED` mmap fails `EINVAL` under the default no-cache +mode (read-only mmap -- git's main use -- works); `O_TMPFILE` is +unsupported (`EOPNOTSUPP`); there is no *remote*-change notification in +the protocol at all (inotify for the guest's own operations works +normally), which is also why `cache=loose` can serve stale data +indefinitely. + +**vng's `--rwdir` shares are not uid-faithful** (measured on the +runner's exact stack): writes arrive on the host as the sharing +daemon's identity -- root when the backend runs under sudo -- and a +non-root guest process gets `EACCES` inside a directory it just +"successfully" created. This is why the git target's +`t/test-results/**` travels via the backend's `--copy-out` (through +the 9p export itself, ownership normalized to the invoker) rather than +via `--share-rw`, and why the source tree's in-suite writes +(`chainlinttmp`, `test-results`) are left to the guest's uid-faithful +tmpfs overlay instead of a share. + +**The guest is disposable; the log is not.** Everything the suite and +the guest console print is teed to `/var/log/eval-under-9p-virtio.log` +on the host (stage2 appends the guest `dmesg` tail on failure), because +by the time `dump-failure-logs.sh` runs, the guest -- and the backing +dir, via teardown -- are gone. vng quirk worth knowing: exit **255** +is also its "guest died before reporting" sentinel, so a suite exiting +255 is indistinguishable from a crash; and exit **124** under CI is the +backend's own `--vm-timeout` catching a boot/mount hang that the +in-guest per-target timeout cannot see. + ## Known-red cells A red cell here is a finding, not a bug report against this repo. These @@ -180,6 +265,61 @@ layers on top, or in the syscalls the pjdfstest column is flagging. Pre-dates the matrix; ext4 is the control row, so this one *is* a real bug worth chasing rather than a filesystem property. +### `9p * / git-annex test` -- hangs, on both servers identically + +The suite's very first group (`testremote type git`) fails `init` +within seconds ("git-annex: Not initialized"), then the run goes +silent inside `unavailable remote / removeKey` until the per-target +timeout kills it (exit 124) ~40 minutes later. Identical on diod and +on QEMU virtfs, so it is v9fs-client/protocol semantics rather than a +server quirk -- and diod's own docs warn that distributed record +locking "will deadlock", which is the pre-registered suspect. A hang +*is* the finding: git-annex on a real vagrant 9p share stalls the same +way. Not yet broken down further; the cell burns its full timeout by +design (a runaway suite reports as `timeout`, not a bare cancellation). + +### `9p * / stress-ng` -- the unlinked-open-file class, server-dependent + +Both rows run all 20 stressors to a clean tally; the failures are +`fstat`/`ftruncate` returning **ENOENT on files that are open but +unlinked** (stress-ng's create-unlink-keep-fd pattern): + +- diod: failed `fallocate`, `hdd`, `copy-file` (17 passed); +- QEMU virtfs: failed `copy-file`, `hdd` -- **`fallocate` passes**, a + clean server divergence worth keeping both rows for. + +On the QEMU row the guest kernel (pinned v6.8) also logs WARN traces +in `v9fs_fid_lookup_with_uid -> v9fs_vfs_getattr_dotl` while these +stressors run -- the client side of the same fid-on-unlinked-file gap, +and part of why the guest kernel is pinned: this signature is +kernel-version-specific. + +### `9p * / pjdfstest` + +Runs to completion on both rows (238 files, ~8800 assertions, ~3 min). +On diod the divergence concentrates in: every `*/03.t` (the +long-pathname scripts) across chown/ftruncate/link/mkdir/mkfifo/mknod/ +open/rmdir/symlink/truncate/unlink; `mkdir|mkfifo|mknod|open/00.t` +assertions 25-27; the `open/06.t` flags matrix (62 of 144); +`rename/10.t` (6 of 2099); `unlink/14.t` #4 (the same assertion the +NFS row flags); `utimensat/08.t`. The QEMU-virtfs row's tally is in +its `logs-*` artifact; per-assertion breakdown not yet done (same +status as the BeeGFS rows). + +### `9p * / git testsuite` + +The full `t0*.sh t1*.sh` selection runs under prove on both rows and +ends with totals; per-script `.out` files are in the artifacts (on the +virtio row they travel out of the guest via the backend's +`--copy-out`). Roll-up: diod 12 scripts / 194 failed assertions, QEMU +virtfs 10 / 177. Shared failures include `t1517-outside-repo` (104) +and `t0450-txt-doc-vs-help` (51) -- both green on the ext4 control +row, so 9p-related, not yet run down. **`t1050-large` (15 failures) +is diod-only** -- consistent with diod's 64 KiB msize cap and the +unlinked-file gap above; it passes under QEMU virtfs. Several of the +remaining entries are `# TODO known breakage` noise the dump's +`not ok` count includes. + ## Red that is not a finding Distinct from the cells above: these are harness races, and the fix is in @@ -228,6 +368,24 @@ check the mount actually came up. `actimeo=0`), locking (`lock` vs `nolock`, and whether `rpc.statd` is even up), `sync` vs `async` on the export, and squashing. Each is a plausible row of its own. +- **The actual vagrant-libvirt 9p default.** vagrant-libvirt's default + `accessmode` is `passthrough`, and distro libvirt runs QEMU as an + unprivileged user -- so the out-of-the-box vagrant 9p share is + *passthrough served by a QEMU that cannot chown*, the very + configuration behind the classic "permission surprises". That is a + third variant, distinct from both the current `mapped` row and + passthrough-as-root. Reproduce it locally today: + `bin/eval-under-9p-virtio --security-model passthrough` run + *without* sudo. A `9p-virtio / passthrough` CI row (root QEMU) is the + cheap second row; the unprivileged flavour needs a small backend + tweak to skip sudo on that path. +- **9p cache/msize variants.** `--cache loose` (the vagrant stale-read + foot-gun) and a small-`msize` row are one matrix line each once the + base rows have settled. +- **virtiofs.** The designated successor to virtio-9p in the same + vagrant/QEMU role (`type: "virtiofs"`), and the natural control row + for "is this 9p, or any VM-shared filesystem?". Nearly free once the + vng plumbing exists: same guest, different device and server. - **Per-assertion breakdown** of the BeeGFS pjdfstest failures. - **A `Loop vfat` mask variant**, if we ever want to separate "vfat is not POSIX" from "vfat mounted with defaults is not POSIX". diff --git a/README.md b/README.md index 5101593..ed4e79f 100644 --- a/README.md +++ b/README.md @@ -30,10 +30,12 @@ itself is both backend- and suite-agnostic: new filesystems drop in as | NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | | Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | | Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | +| 9p tcp (diod) | [![9p tcp (diod) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-git-annex.svg)](https://con.github.io/eval-under/#9p-tcp-git-annex) | [![9p tcp (diod) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-git.svg)](https://con.github.io/eval-under/#9p-tcp-git) | [![9p tcp (diod) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-stress-ng.svg)](https://con.github.io/eval-under/#9p-tcp-stress-ng) | [![9p tcp (diod) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-tcp-pjdfstest.svg)](https://con.github.io/eval-under/#9p-tcp-pjdfstest) | +| 9p virtio (mapped) | [![9p virtio (mapped) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-git-annex.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-git-annex) | [![9p virtio (mapped) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-git.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-git) | [![9p virtio (mapped) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-stress-ng.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-stress-ng) | [![9p virtio (mapped) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/9p-virtio-mapped-pjdfstest.svg)](https://con.github.io/eval-under/#9p-virtio-mapped-pjdfstest) | Rows are **backends** (which filesystem the work happens on), columns -are **targets** (which suite is run on it). All 20 cells are one job +are **targets** (which suite is run on it). All 28 cells are one job matrix in [`.github/workflows/test.yaml`](.github/workflows/test.yaml), fanned out from [`.github/matrix.yaml`](.github/matrix.yaml) -- adding a filesystem or a suite is a data edit, not a code edit. @@ -145,6 +147,20 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \ # the fsync-heavy slow path) sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test' +# Under 9p served by diod over localhost TCP (the no-VM 9p; root is +# only used for the mount -- the server and the command run as you) +sudo bin/eval-under 9p-tcp --set-home -- bash -c 'cd "$HOME" && git annex test' + +# Under the real Vagrant/QEMU virtio-9p stack: the command runs INSIDE +# a virtme-ng guest booted from this host's own root filesystem. Needs +# no sudo at all (KVM access aside); --cache loose reproduces the +# classic vagrant stale-read foot-gun. +bin/eval-under 9p-virtio --set-home -- bash -c 'cd "$HOME" && git annex test' +bin/eval-under 9p-virtio --cache loose -- some-command + +# Poke around inside a live guest with the 9p mount up +bin/eval-under 9p-virtio --shell + # Skip teardown to poke around after a failure sudo bin/eval-under beegfs --set-home --keep -- some-failing-command @@ -166,6 +182,8 @@ the full flag / env-var / default table per backend. | `bin/eval-under-beegfs` | BeeGFS backend (containerised cluster + kernel client mount) | | `bin/eval-under-nfs` | NFS backend (localhost loopback export) | | `bin/eval-under-loop` | Loop-device backend (dd + losetup + mkfs. + mount) | +| `bin/eval-under-9p-tcp` | 9p backend, no VM: diod (9P2000.L) on localhost TCP + kernel v9fs mount | +| `bin/eval-under-9p-virtio` | 9p backend, the Vagrant/QEMU stack: QEMU -virtfs into a virtme-ng guest | | `fixtures/beegfs/docker-compose-v7.yml` | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host` | | `fixtures/beegfs/docker-compose-v8.yml` | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane) | | `fixtures/beegfs/beegfs-*.conf.template` | Minimal client + helperd confs for the throwaway cluster | @@ -179,7 +197,7 @@ the full flag / env-var / default table per backend. | `bin/ci/update-status.py` | Merges a run's per-cell results into the persistent `status.json` | | `bin/ci/render-report.py` | Renders `status.json` into the badge set + the report page | | `bin/ci/publish-status.sh` | Ties those together and pushes the site to `gh-pages` | -| `.github/workflows/test.yaml` | The whole matrix: one `matrix` job, 20 `test` cells, one `publish` job | +| `.github/workflows/test.yaml` | The whole matrix: one `matrix` job, 28 `test` cells, one `publish` job | | `drafts/git-annex-test-beegfs.yaml` | Copy-target workflow for `con/git-annex` (external PR target) | ## Local iteration (VM) @@ -198,6 +216,8 @@ sudo bin/eval-under beegfs --set-home -- bash -c ' ' sudo bin/eval-under nfs --set-home -- git annex test sudo bin/eval-under loop --fs vfat --set-home -- git annex test +sudo bin/eval-under 9p-tcp --set-home -- git annex test +bin/eval-under 9p-virtio --set-home -- git annex test # nested KVM is on # Or run a whole CI cell exactly as the runner would. install-target.sh # is the one-off prep (source builds land in $EVAL_UNDER_SRC_DIR, not on @@ -205,6 +225,8 @@ sudo bin/eval-under loop --fs vfat --set-home -- git annex test bin/ci/install-target.sh pjdfstest sudo -E bin/ci/run-under.sh loop ext4 pjdfstest sudo -E bin/ci/run-under.sh nfs n/a stress-ng +sudo -E bin/ci/run-under.sh 9p-tcp n/a pjdfstest +sudo -E bin/ci/run-under.sh 9p-virtio mapped stress-ng ``` Optional: install `act` in the VM to replay the GitHub workflow locally. diff --git a/bin/ci/dump-failure-logs.sh b/bin/ci/dump-failure-logs.sh index 6253b18..6317dcb 100755 --- a/bin/ci/dump-failure-logs.sh +++ b/bin/ci/dump-failure-logs.sh @@ -12,8 +12,12 @@ # bin/ci/dump-failure-logs.sh [target] # # backend side: -# beegfs: `docker compose logs` + dmesg-filtered-for-beegfs -# nfs/loop: full dmesg tail +# beegfs: `docker compose logs` + dmesg-filtered-for-beegfs +# nfs/loop: filtered dmesg tail +# 9p-tcp: filtered dmesg tail (the v9fs client logs there) +# 9p-virtio: tail of the guest console/suite log the backend tees to +# /var/log/eval-under-9p-virtio.log (guest dmesg included +# on failure; the guest itself is gone by now) # target side: # git: the failing assertions + output from t/test-results/*.out @@ -37,7 +41,7 @@ case "$BACKEND" in echo "=== dmesg (beegfs-tagged, last 50) ===" sudo dmesg | grep -i beegfs | tail -50 || true ;; - nfs|loop) + nfs|loop|9p-tcp) # Filtered rather than `dmesg | tail -100`: on a hosted runner the # last 100 kernel lines are almost entirely boot spam (hyperv, pci, # apparmor), which buries the failure in the job log. Warnings and @@ -47,9 +51,21 @@ case "$BACKEND" in sudo dmesg --level=emerg,alert,crit,err,warn 2>/dev/null | tail -40 || true echo "=== dmesg (mentioning $BACKEND/$VERSION, last 30) ===" sudo dmesg 2>/dev/null \ - | grep -iE "loop|nfs|${VERSION:-nomatch}" \ + | grep -iE "loop|nfs|9p|${VERSION:-nomatch}" \ | tail -30 || true ;; + 9p-virtio) + # The suite ran inside a virtme-ng guest that no longer exists; + # what survives is the console/suite log the backend tees on the + # host (stage2 appends the guest dmesg tail there on failure). + log="${EVAL_UNDER_9P_VIRTIO_LOG:-/var/log/eval-under-9p-virtio.log}" + echo "=== 9p-virtio guest log (last 120 lines of $log) ===" + if [ -r "$log" ]; then + tail -120 "$log" || true + else + echo "no log at $log (the guest never started?)" + fi + ;; *) echo "unknown backend: $BACKEND" >&2 ;; diff --git a/bin/ci/gen-readme-matrix.sh b/bin/ci/gen-readme-matrix.sh index ed3d5b8..94c6a62 100755 --- a/bin/ci/gen-readme-matrix.sh +++ b/bin/ci/gen-readme-matrix.sh @@ -72,7 +72,7 @@ table_md() { printf '\n' for cell in "${EVAL_UNDER_BACKENDS[@]}"; do - IFS='|' read -r backend version label <<< "$cell" + IFS='|' read -r backend version label _runs_on <<< "$cell" printf '| %s |' "$label" for target in "${EVAL_UNDER_TARGETS[@]}"; do slug="$(cell_slug "$backend" "$version" "$target")" diff --git a/bin/ci/install-backend.sh b/bin/ci/install-backend.sh index 6c5ee54..ae329c5 100755 --- a/bin/ci/install-backend.sh +++ b/bin/ci/install-backend.sh @@ -9,18 +9,27 @@ # usage: # bin/ci/install-backend.sh # -# backend = beegfs | nfs | loop -# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) -# for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) -# for nfs: literal "n/a" +# backend = beegfs | nfs | loop | 9p-tcp | 9p-virtio +# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) +# for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) +# for nfs: literal "n/a" +# for 9p-tcp: literal "n/a" +# for 9p-virtio: security-model row (mapped, passthrough) -- +# same packages either way # # Idempotent enough for CI re-runs; not a full package manager. set -euo pipefail export DEBIAN_FRONTEND=noninteractive -BACKEND="${1:?backend required (beegfs|nfs|loop)}" -VERSION="${2:?version required (BeeGFS version | loop fs name | 'n/a' for nfs)}" +here="$(cd "$(dirname "$0")" && pwd)" +# matrix.sh is a sourced library (pinned refs, e.g. the 9p guest +# kernel), resolved at runtime relative to $here. +# shellcheck source=bin/ci/matrix.sh disable=SC1091 +. "$here/matrix.sh" + +BACKEND="${1:?backend required (beegfs|nfs|loop|9p-tcp|9p-virtio)}" +VERSION="${2:?version required (BeeGFS version | loop fs name | 9p variant | 'n/a')}" # Give unattended-upgrades a moment on ubuntu-22.04 runners rather than # hard-failing on a dpkg lock. @@ -86,9 +95,56 @@ install_loop() { command -v "mkfs.$VERSION" } +install_9p_tcp() { + apt_update + apt_install diod + command -v diod + # 9p client modules ship in the kernel's base linux-modules package + # on generic, virtual and azure flavours alike -- no modules-extra + # needed. Load and prove the filesystem is registered now, so a + # missing module is an install-step error, not a mid-suite one. + sudo modprobe -a 9p 9pnet_fd + grep -w 9p /proc/filesystems +} + +install_9p_virtio() { + apt_update + # virtme-ng only Recommends qemu and Suggests virtiofsd, and this + # script installs with --no-install-recommends -- list everything + # explicitly. busybox-static: vng builds an initramfs for a pinned + # --run kernel and finds no busybox on the runner images otherwise. + # zstd: the mainline kernel debs ship .ko.zst modules and vng shells + # out to zstd for them (a guest that dies with vng's 255 sentinel + # after "zstd: not found" is this). Without virtiofsd, vng silently + # serves the rootfs over 9p instead (works, but slower and it + # shares the transport under test). + apt_install qemu-system-x86 qemu-utils virtme-ng virtiofsd busybox-static zstd + vng --version + + # KVM node permissions for the unprivileged prewarm below (the + # run-under.sh invocation is root via sudo -E and needs only the + # node to exist). Standard GitHub Linux runners have had KVM since + # 2024; this is the documented enablement. + echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ + | sudo tee /etc/udev/rules.d/99-kvm4all.rules >/dev/null + sudo udevadm control --reload-rules + sudo udevadm trigger --name-match=kvm + ls -l /dev/kvm + + # Pre-warm outside the suite's timeout: the pinned mainline kernel + # is a ~180 MB one-time download into ~/.cache/virtme-ng (the + # sudo -E run later resolves to the same HOME), and this doubles as + # a boot smoke test -- a vng/KVM problem fails the install step + # with its own logs instead of a red cell mid-matrix. + timeout 600 vng --run "$EVAL_UNDER_9P_KERNEL_REF" --disable-microvm \ + --memory 1G -- true +} + case "$BACKEND" in - beegfs) install_beegfs ;; - nfs) install_nfs ;; - loop) install_loop ;; - *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop)" >&2; exit 1 ;; + beegfs) install_beegfs ;; + nfs) install_nfs ;; + loop) install_loop ;; + 9p-tcp) install_9p_tcp ;; + 9p-virtio) install_9p_virtio ;; + *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop|9p-tcp|9p-virtio)" >&2; exit 1 ;; esac diff --git a/bin/ci/matrix-json.sh b/bin/ci/matrix-json.sh index 6c1ca13..bd48de6 100755 --- a/bin/ci/matrix-json.sh +++ b/bin/ci/matrix-json.sh @@ -14,10 +14,12 @@ # name job name, e.g. "BeeGFS 7.4.6 / git testsuite". Set as the # job's `name:` so the checks list reads like the README grid # instead of GitHub's default "test (beegfs, 7.4.6, git)". -# backend eval-under backend (beegfs | nfs | loop) +# backend eval-under backend (beegfs | nfs | loop | 9p-*) # version backend version, or "n/a" # target suite to run under it # slug filename-safe cell id, for artifact names +# runs-on runner image for this cell (per-backend in matrix.yaml, +# default ubuntu-22.04) # # usage: # bin/ci/matrix-json.sh # all cells @@ -34,9 +36,9 @@ here="$(cd "$(dirname "$0")" && pwd)" entries=() for cell in "${EVAL_UNDER_BACKENDS[@]}"; do - IFS='|' read -r backend version label <<< "$cell" + IFS='|' read -r backend version label runs_on <<< "$cell" for target in "${EVAL_UNDER_TARGETS[@]}"; do - entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") + entries+=("$backend|$version|$label|$runs_on|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") done done @@ -48,13 +50,14 @@ for line in sys.stdin: line = line.rstrip("\n") if not line: continue - backend, version, blabel, target, tlabel, slug = line.split("|") + backend, version, blabel, runs_on, target, tlabel, slug = line.split("|") include.append({ "name": "%s / %s" % (blabel, tlabel), "backend": backend, "version": version, "target": target, "slug": slug, + "runs-on": runs_on, }) if not include: diff --git a/bin/ci/matrix.sh b/bin/ci/matrix.sh index 9a2dd70..9320865 100755 --- a/bin/ci/matrix.sh +++ b/bin/ci/matrix.sh @@ -51,8 +51,13 @@ def q(v): out = [] out.append("EVAL_UNDER_TARGETS=(%s)" % " ".join(q(t["name"]) for t in targets)) +# Four-field tuple; consumers destructure with +# `IFS='|' read -r backend version label runs_on`. runs-on is optional +# in the YAML (default ubuntu-22.04) so existing rows stay untouched. out.append("EVAL_UNDER_BACKENDS=(%s)" % " ".join( - q("%s|%s|%s" % (b["backend"], b["version"], b["label"])) for b in backends)) + q("%s|%s|%s|%s" % (b["backend"], b["version"], b["label"], + b.get("runs-on", "ubuntu-22.04"))) + for b in backends)) out.append("declare -A _EU_LABEL=(%s)" % " ".join( "[%s]=%s" % (q(t["name"]), q(t["label"])) for t in targets)) @@ -70,6 +75,7 @@ out.append(": \"${EVAL_UNDER_REPO_SLUG:=%s}\"" % q(d["repo-slug"])) out.append(": \"${EVAL_UNDER_SRC_DIR:=%s}\"" % q(d["src-dir"])) out.append(": \"${EVAL_UNDER_GIT_REF:=%s}\"" % q(d["refs"]["git"])) out.append(": \"${EVAL_UNDER_PJDFSTEST_REF:=%s}\"" % q(d["refs"]["pjdfstest"])) +out.append(": \"${EVAL_UNDER_9P_KERNEL_REF:=%s}\"" % q(d["refs"]["9p-kernel"])) print("\n".join(out)) PYEOF @@ -83,7 +89,7 @@ PYEOF } export EVAL_UNDER_REPO_SLUG EVAL_UNDER_SRC_DIR -export EVAL_UNDER_GIT_REF EVAL_UNDER_PJDFSTEST_REF +export EVAL_UNDER_GIT_REF EVAL_UNDER_PJDFSTEST_REF EVAL_UNDER_9P_KERNEL_REF # Filename-safe identifier for a backend cell: "beegfs-7.4.6", "nfs", # "loop-vfat". diff --git a/bin/ci/render-badge.sh b/bin/ci/render-badge.sh index 4bf8ef2..db50a15 100755 --- a/bin/ci/render-badge.sh +++ b/bin/ci/render-badge.sh @@ -7,10 +7,10 @@ # Render one status badge as a self-contained SVG on stdout. # # We draw these ourselves rather than linking shields.io endpoint -# badges: the grid is 20 cells, so a README render would otherwise be -# 20 third-party requests, and the badges would go blank whenever that -# service is unreachable. An SVG committed to the `badges` branch has -# neither problem and is diffable. +# badges: the grid is dozens of cells, so a README render would +# otherwise be dozens of third-party requests, and the badges would go +# blank whenever that service is unreachable. An SVG committed to the +# `badges` branch has neither problem and is diffable. # # Single-segment on purpose: in the README grid the row and column # headers already name the cell, so a "BeeGFS 7.4.6 / git" prefix on diff --git a/bin/ci/run-under.sh b/bin/ci/run-under.sh index 8580891..6dbbeb7 100755 --- a/bin/ci/run-under.sh +++ b/bin/ci/run-under.sh @@ -13,10 +13,13 @@ # usage: # bin/ci/run-under.sh [target] # -# backend = beegfs | nfs | loop -# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) -# for loop: filesystem type (e.g. vfat, ext4) -# for nfs: literal "n/a" +# backend = beegfs | nfs | loop | 9p-tcp | 9p-virtio +# version = for beegfs: point release (e.g. 7.4.6, 8.1.0) +# for loop: filesystem type (e.g. vfat, ext4) +# for nfs: literal "n/a" +# for 9p-tcp: literal "n/a" +# for 9p-virtio: QEMU virtfs security model row +# (mapped -> mapped-xattr, passthrough) # target = git-annex (default) | git | stress-ng | pjdfstest # # env overrides: @@ -59,6 +62,36 @@ case "$BACKEND" in # need an export that does not squash root, and need to keep # their privileges rather than being dropped to the invoker. target_needs_root "$TARGET" && opts=(--no-root-squash) ;; + 9p-tcp) + opts=() + # Same shape as NFS: the default is diod's single-user mode + # with the command dropped to the invoker; root-requiring + # suites need the multi-user export and their privileges. + target_needs_root "$TARGET" && opts=(--run-as-root) ;; + 9p-virtio) + # The version token is the QEMU virtfs security-model row. + case "$VERSION" in + mapped) secmodel=mapped-xattr ;; + passthrough) secmodel=passthrough ;; + *) echo "unknown 9p-virtio variant: $VERSION" >&2; exit 1 ;; + esac + # Pinned guest kernel: handed over explicitly, so the backend + # stays matrix-free and defaults to the host kernel locally. + # The VM timeout guards boot/mount hangs the in-guest + # per-target timeout cannot see; memory is CI-sized (the + # script's own default suits laptops). + opts=(--security-model "$secmodel" + --kernel "$EVAL_UNDER_9P_KERNEL_REF" + --memory 4G + --vm-timeout $((TIMEOUT + 300))) + # git writes its per-script logs into the source tree; + # inside the guest that tree is a discardable overlay (vng + # --rwdir is deliberately NOT used: it is not uid-faithful, + # see the backend's --share-rw caveat), so ferry the + # results back through the 9p export instead for the + # artifact upload. + [ "$TARGET" = git ] && opts+=(--copy-out "$EVAL_UNDER_SRC_DIR/git/t/test-results") + target_needs_root "$TARGET" && opts+=(--run-as-root) ;; *) echo "unknown backend: $BACKEND" >&2; exit 1 ;; esac diff --git a/bin/ci/update-status.py b/bin/ci/update-status.py index 40b77b1..316ea07 100755 --- a/bin/ci/update-status.py +++ b/bin/ci/update-status.py @@ -9,10 +9,11 @@ # # Why a persistent file rather than just reading the current run: a run # does not necessarily cover every cell. GitHub's "Re-run failed jobs" -# re-executes only the red ones, so a run's artifacts can describe 10 of -# 20 cells. Deriving the whole grid from one run would rewrite the other -# 10 badges to "unknown" and destroy good state. So results are merged, -# and a cell absent from this run keeps whatever it last reported. +# re-executes only the red ones, so a run's artifacts can describe a +# fraction of the grid. Deriving the whole grid from one run would +# rewrite every other badge to "unknown" and destroy good state. So +# results are merged, and a cell absent from this run keeps whatever it +# last reported. # # (This is a deliberate divergence from con/git-annex's update.py, which # sets absent tests to UNKNOWN. There, a client that stops reporting a diff --git a/bin/eval-under-9p-tcp b/bin/eval-under-9p-tcp new file mode 100755 index 0000000..49dae48 --- /dev/null +++ b/bin/eval-under-9p-tcp @@ -0,0 +1,310 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-9p-tcp: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR (and +# optionally HOME) pointing at a 9p mount served over TCP by diod (LLNL's +# 9P2000.L server) on localhost -- same architecture as eval-under-nfs +# (local server + kernel client), different protocol and server. +# +# This exercises the kernel v9fs client against diod's server semantics. +# For the QEMU/virtio-9p stack that Vagrant synced folders actually use, +# see eval-under-9p-virtio; the v9fs client code is the same, the server +# (and its locking/caching quirks) is not. + +set -eu + +# Defaults (options override; env vars override defaults). +MNT="${EVAL_UNDER_MOUNT:-/mnt/9p}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +PORT="${EVAL_UNDER_9P_TCP_PORT:-5640}" +MSIZE="${EVAL_UNDER_9P_TCP_MSIZE:-}" +CACHE="${EVAL_UNDER_9P_TCP_CACHE:-}" +RUN_AS_ROOT="${EVAL_UNDER_9P_TCP_RUN_AS_ROOT:-0}" +STARTUP_WAIT="${EVAL_UNDER_9P_TCP_STARTUP_WAIT:-10}" + +usage() { + cat <<'EOF' +Usage: eval-under-9p-tcp [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at a 9p mount. A fresh backing directory is exported on +127.0.0.1 by diod (9P2000.L) and mounted via `mount -t 9p -o trans=tcp` +for the duration of the command. + +Requires the `diod` package (Ubuntu/Debian universe) and a kernel with +9p client support (CONFIG_9P_FS; in the stock `linux-modules` package +of Ubuntu generic/virtual/azure kernels). + +By default diod itself runs unprivileged AS THE INVOKING USER in its +documented single-user mode (mount options uname=,access=), +and the wrapped command runs as that user too -- root is used only for +the mount/umount pair. --run-as-root switches to diod's multi-user mode +(server as root, access=client) and keeps the wrapped command as root, +for suites that measure privileged behaviour. + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT /mnt/9p + Where to mount the export on the host. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Skip teardown; leave diod + mount up for debugging. + + --port P EVAL_UNDER_9P_TCP_PORT 5640 + TCP port for diod on 127.0.0.1 (unprivileged; not 564 so nothing + collides with a system-wide 9p service). + + --msize BYTES EVAL_UNDER_9P_TCP_MSIZE (unset) + Client request size. Unset = kernel default (128 KiB since 5.15) + -- which diod then negotiates down to its own 64 KiB cap. + + --cache MODE EVAL_UNDER_9P_TCP_CACHE (unset) + v9fs cache mode (none|loose|fscache|mmap|...; readahead/writeback + on kernels >= 6.4). Unset = kernel default, which is no caching in + every kernel era. `loose` is the classic stale-read foot-gun. + + --run-as-root EVAL_UNDER_9P_TCP_RUN_AS_ROOT (unset) + Run diod as root in multi-user mode (uname=root,access=client) + and keep the wrapped command running as root. Needed for suites + that measure privileged behaviour (pjdfstest, stress-ng's + chown/mknod stressors) -- the 9p analog of eval-under-nfs's + --no-root-squash. + + --startup-wait S EVAL_UNDER_9P_TCP_STARTUP_WAIT 10 + Seconds to wait for diod to bind its port before giving up. + + -h, --help + Print this help. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + +Exit status: the wrapped command's exit status. Teardown runs on any +exit (unless --keep). +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --port) PORT="$2"; shift 2 ;; + --msize) MSIZE="$2"; shift 2 ;; + --cache) CACHE="$2"; shift 2 ;; + --run-as-root) RUN_AS_ROOT=1; shift ;; + --startup-wait) STARTUP_WAIT="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ $# -gt 0 ] || { echo "no command given" >&2; usage >&2; exit 2; } + +if [ "$(id -u)" -ne 0 ]; then + command -v sudo >/dev/null || { + echo "must run as root (or have sudo available)" >&2; exit 2; } + SUDO=(sudo) +else + SUDO=() +fi + +# Identify the invoking (non-root) user: diod's single-user mode serves +# exactly one uid, and the wrapped command should run as the pre-sudo +# user, mirroring eval-under-nfs's root_squash default. +if [ -n "${SUDO_UID:-}" ]; then + INVOKER_UID="$SUDO_UID"; INVOKER_GID="${SUDO_GID:-$SUDO_UID}" + INVOKER_USER="${SUDO_USER:-$INVOKER_UID}" +else + INVOKER_UID="$(id -u)"; INVOKER_GID="$(id -g)" + INVOKER_USER="$(id -un)" +fi + +log() { printf '\nI: %s\n' "$*"; } + +ensure_9p_installed() { + command -v diod >/dev/null 2>&1 || { + echo "ERROR: diod not installed." >&2 + echo " Install with: apt install diod" >&2 + exit 3 + } + # 9p client modules live in the kernel's base linux-modules package + # (generic, virtual and azure flavours alike); modprobe is only needed + # because nothing else has loaded them yet. trans=tcp itself lives in + # 9pnet_fd (its own module since ~5.17), autoloaded on mount. + "${SUDO[@]}" modprobe -q 9p 2>/dev/null || true + grep -qw 9p /proc/filesystems || { + echo "ERROR: kernel has no 9p filesystem support (CONFIG_9P_FS)." >&2 + echo " On Ubuntu the modules ship in linux-modules-\$(uname -r);" >&2 + echo " a container kernel without them cannot run this backend." >&2 + exit 3 + } +} + +# Backing dir that gets exported (distinct from the mountpoint so we +# don't 9p-loop-back into ourselves). +ORIG="$(mktemp -u "${TMPDIR:-/tmp}/eval-under-9p-XXXXX").orig" +DIOD_LOG="$(mktemp "${TMPDIR:-/tmp}/eval-under-9p-diod-XXXXX.log")" +DIOD_PID="" + +start_diod() { + log "creating backing dir $ORIG (owned by uid=$INVOKER_UID gid=$INVOKER_GID)" + "${SUDO[@]}" mkdir -p "$ORIG" + "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$ORIG" + + # The log stays owned by whoever this script runs as: the redirects + # below are performed by the current shell and diod only inherits the + # open fd, so diod's own uid never needs write access to the file. + # (Do NOT chown it to the invoker: with the script running as root, + # fs.protected_regular then blocks root's O_CREAT append in sticky + # /tmp -- "Permission denied" before diod ever starts.) + # + # --foreground so it stays our child (we background + kill it + # ourselves); --no-auth because munge has no place in a loopback + # throwaway export. + if [ "$RUN_AS_ROOT" != 0 ]; then + log "starting diod as root (multi-user): 127.0.0.1:$PORT <- $ORIG" + "${SUDO[@]}" diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + elif [ "$INVOKER_UID" != "$(id -u)" ]; then + log "starting diod as $INVOKER_USER (single-user): 127.0.0.1:$PORT <- $ORIG" + # The redirect deliberately happens in the current (root) shell, not + # under sudo: the shell owns the log and opens the fd, diod merely + # inherits it. + # shellcheck disable=SC2024 + sudo -u "$INVOKER_USER" diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + else + log "starting diod as $(id -un) (single-user): 127.0.0.1:$PORT <- $ORIG" + diod --foreground --no-auth \ + --listen "127.0.0.1:$PORT" --export "$ORIG" >>"$DIOD_LOG" 2>&1 & + fi + DIOD_PID=$! + + local waited=0 + while [ "$waited" -lt "$STARTUP_WAIT" ]; do + if ! kill -0 "$DIOD_PID" 2>/dev/null; then + break # died; report below + fi + if "${SUDO[@]}" ss -ltn 2>/dev/null | awk '{print $4}' | grep -qx "127.0.0.1:$PORT"; then + log "diod is listening (pid $DIOD_PID, after ${waited}s)" + return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + echo "ERROR: diod did not bind 127.0.0.1:$PORT within ${STARTUP_WAIT}s" >&2 + sed 's/^/ diod: /' "$DIOD_LOG" >&2 || true + exit 4 +} + +mount_9p() { + local mopts + # Option pairings are diod's documented ones: single-user mode serves + # exactly the invoking uid (uname=,access=); multi-user + # mode is the NFS-like uname=root,access=client. + if [ "$RUN_AS_ROOT" != 0 ]; then + mopts="trans=tcp,port=$PORT,version=9p2000.L,aname=$ORIG,uname=root,access=client" + else + mopts="trans=tcp,port=$PORT,version=9p2000.L,aname=$ORIG,uname=$INVOKER_USER,access=$INVOKER_UID" + fi + [ -n "$MSIZE" ] && mopts="$mopts,msize=$MSIZE" + [ -n "$CACHE" ] && mopts="$mopts,cache=$CACHE" + + log "mounting 9p at $MNT (-o $mopts)" + "${SUDO[@]}" mkdir -p "$MNT" + "${SUDO[@]}" mount -t 9p -o "$mopts" 127.0.0.1 "$MNT" + # The /proc/mounts line records the *effective* options -- diod caps + # msize at 64 KiB whatever the client asked for, and GOTCHAS.md wants + # settings as measured, not as requested. + grep -F " $MNT " /proc/mounts | sed 's/^/I: /' || true +} + +# mount(2) returning is not the same as the export being usable; probe +# with a real create + write + read-back, as the invoker when access= is +# restricted to that uid (root is not exempt from v9fs access checks). +wait_for_mount_usable() { + local probe="$MNT/.eval-under-ready.$$" + local as_user=() + if [ "$RUN_AS_ROOT" = 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + as_user=(sudo -u "$INVOKER_USER") + fi + local waited=0 err="" + while [ "$waited" -lt 10 ]; do + if err=$("${as_user[@]}" dd if=/dev/urandom of="$probe" bs=4k count=1 2>&1) \ + && err=$("${as_user[@]}" dd if="$probe" of=/dev/null bs=4k count=1 2>&1); then + "${as_user[@]}" rm -f "$probe" 2>/dev/null || true + log "mount is usable (after ${waited}s)" + return 0 + fi + sleep 1 + waited=$((waited + 1)) + done + echo "ERROR: 9p mounted at $MNT but unusable after 10s" >&2 + echo " last probe error: $err" >&2 + sed 's/^/ diod: /' "$DIOD_LOG" >&2 || true + "${SUDO[@]}" dmesg 2>/dev/null | grep -i 9p | tail -20 >&2 || true + exit 6 +} + +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving diod (pid $DIOD_PID) + mount up (exit=$rc)" + echo "I: diod log: $DIOD_LOG" + return "$rc" + fi + log "teardown" + "${SUDO[@]}" umount "$MNT" 2>/dev/null || true + if [ -n "$DIOD_PID" ]; then + "${SUDO[@]}" kill "$DIOD_PID" 2>/dev/null || true + fi + "${SUDO[@]}" rm -rf "$MNT" "$ORIG" "$DIOD_LOG" 2>/dev/null || true + return "$rc" +} +trap teardown EXIT + +ensure_9p_installed +start_diod +mount_9p +wait_for_mount_usable + +RUN_HOME="$HOME" +if [ "$SET_HOME" = 1 ]; then + RUN_HOME="$MNT/home" + # mkdir as the invoker -- under access= a root mkdir would be + # rejected by the client, same shape as NFS root_squash. + if [ "$RUN_AS_ROOT" = 0 ] \ + && [ "$INVOKER_UID" != 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + sudo -u "$INVOKER_USER" mkdir -p "$RUN_HOME" + else + mkdir -p "$RUN_HOME" + fi +fi + +# Drop back to the invoking user for the wrapped command (single-user +# diod serves only that uid anyway); --run-as-root keeps privileges to +# match its multi-user export. Mirrors eval-under-nfs. +if [ "$RUN_AS_ROOT" = 0 ] \ + && [ "$INVOKER_UID" != 0 ] && [ "$INVOKER_UID" != "$(id -u)" ]; then + log "running (as uid=$INVOKER_UID): $*" + sudo -u "$INVOKER_USER" -E \ + env HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" \ + "$@" +else + log "running: $*" + HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" "$@" +fi diff --git a/bin/eval-under-9p-virtio b/bin/eval-under-9p-virtio new file mode 100755 index 0000000..e95439a --- /dev/null +++ b/bin/eval-under-9p-virtio @@ -0,0 +1,601 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-9p-virtio: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR +# (and optionally HOME) pointing at a virtio-9p mount -- the actual stack +# behind Vagrant/libvirt "9p" synced folders, WSL2 drive shares, and +# QEMU -virtfs in general: QEMU's virtfs server on the host, the kernel +# v9fs client in a guest, virtio in between. +# +# The guest comes from virtme-ng (vng): it boots a QEMU guest whose root +# filesystem is a read-only view of the host's own root (with tmpfs +# overlays), so every tool installed on the host -- git, git-annex, +# prove, the bin/ci target scripts -- exists in the guest unchanged. +# This script adds one more -virtfs device exporting a fresh scratch +# dir, re-invokes itself inside the guest (--guest-stage2) to mount it +# and run the wrapped command there, and propagates the exit status +# back out (vng carries it over a dedicated virtio-serial channel). +# +# Notes that shape the implementation: +# - vng's rootfs is read-only outside its overlay set (/etc /lib /home +# /opt /srv /usr /var, plus a fresh tmpfs /tmp), so the *guest* +# mountpoint defaults to /tmp/eval-under-9p and --mount-point here +# names a guest path. Writes to overlay paths evaporate with the +# guest; use --share-rw for host dirs that must receive writes. +# - vng's microvm machine has no PCI bus the stock Ubuntu/mainline +# guest kernels can enumerate, so --disable-microvm is passed to +# keep -virtfs (virtio-9p-pci) visible. +# - guest scripts run as root by default; the wrapped command is +# dropped to the invoking user inside the guest (runuser) unless +# --run-as-root, mirroring eval-under-nfs. +# - unlike the other backends this one needs NO root on the host for +# security models other than passthrough-as-root: QEMU, vng and the +# mount (which happens inside the guest) all run unprivileged. + +set -eu + +here="$(cd "$(dirname "$0")" && pwd)" +SELF="$here/$(basename "$0")" + +# Defaults (options override; env vars override defaults). +MNT="${EVAL_UNDER_MOUNT:-/tmp/eval-under-9p}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +SECMODEL="${EVAL_UNDER_9P_VIRTIO_SECURITY_MODEL:-mapped-xattr}" +KERNEL="${EVAL_UNDER_9P_VIRTIO_KERNEL:-host}" +MEMORY="${EVAL_UNDER_9P_VIRTIO_MEMORY:-2G}" +CPUS="${EVAL_UNDER_9P_VIRTIO_CPUS:-$(nproc)}" +MSIZE="${EVAL_UNDER_9P_VIRTIO_MSIZE:-}" +CACHE="${EVAL_UNDER_9P_VIRTIO_CACHE:-}" +WRITEOUT="${EVAL_UNDER_9P_VIRTIO_WRITEOUT:-}" +RUN_AS_ROOT="${EVAL_UNDER_9P_VIRTIO_RUN_AS_ROOT:-0}" +VM_TIMEOUT="${EVAL_UNDER_9P_VIRTIO_VM_TIMEOUT:-0}" +ALLOW_TCG="${EVAL_UNDER_9P_VIRTIO_ALLOW_TCG:-0}" +LOG="${EVAL_UNDER_9P_VIRTIO_LOG:-}" +DEBUG_BOOT="${EVAL_UNDER_9P_VIRTIO_DEBUG_BOOT:-0}" +SHELL_MODE=0 +TAG="eval9p" + +# Space-separated initial values from the environment; the flags append. +read -r -a SHARE_RW <<< "${EVAL_UNDER_9P_VIRTIO_SHARE_RW:-}" +read -r -a COPY_OUT <<< "${EVAL_UNDER_9P_VIRTIO_COPY_OUT:-}" + +usage() { + cat <<'EOF' +Usage: eval-under-9p-virtio [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at a virtio-9p mount: a fresh scratch dir on the host is +exported by QEMU's virtfs server (-virtfs local,...) into a virtme-ng +guest booted from the host's own root filesystem, mounted there with +`mount -t 9p -o trans=virtio`, and CMD runs INSIDE that guest. This is +the stack Vagrant/libvirt "9p" synced folders use. + +Requires: virtme-ng (vng), qemu-system-x86; busybox for a pinned +--kernel (package busybox-static); /dev/kvm strongly recommended. +Install on Ubuntu 24.04: apt install virtme-ng virtiofsd busybox-static +(22.04 has no virtme-ng package: pipx install virtme-ng). + +Unlike the other backends, no host root is needed except for +--security-model passthrough served by a root QEMU (run the whole +script under sudo for that). CI does run it under sudo -E; that is +what makes its passthrough row "passthrough as root". + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT /tmp/eval-under-9p + GUEST path to mount the export on. Must be creatable in the + guest: vng's rootfs is read-only outside /etc /lib /home /opt + /srv /usr /var and a fresh tmpfs /tmp -- the default lives there + on purpose. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Keep the host-side backing dir + log after exit and print how to + relaunch. The guest itself always ends with the command; use + --shell for an interactive look INSIDE a live guest. Note that + under mapped-xattr the kept files read host-side as mode 0600/0700 + owned by the QEMU user, with the real metadata in user.virtfs.* + xattrs. + + --shell (unset) + Instead of CMD, run an interactive bash inside the guest with the + mount already up (best-effort: the console is a virtio-serial + channel, so no job control). The morning-after-a-red-cell tool. + + --security-model M EVAL_UNDER_9P_VIRTIO_SECURITY_MODEL mapped-xattr + QEMU virtfs server model: mapped-xattr (ownership/mode/devices + faked in user.virtfs.* xattrs -- vagrant-libvirt accessmode + "mapped"), mapped-file, passthrough (real uids; wants a root + QEMU -- run this script under sudo), none (passthrough that + ignores chown failures -- vagrant-libvirt "squash"). + NOTE vagrant-libvirt's *default* accessmode is passthrough under + an unprivileged QEMU; reproduce that by running this script + unprivileged with --security-model passthrough. + + --kernel VER EVAL_UNDER_9P_VIRTIO_KERNEL host + Guest kernel. "host" boots the host's own kernel and modules (no + download; /boot/vmlinuz-* must be readable -- Ubuntu ships it + 0600, so chmod +r it or run under sudo). A version tag like + "v6.8" makes vng download that Ubuntu mainline build (~180 MB + once, cached in ~/.cache/virtme-ng) -- what CI uses so the v9fs + client is pinned. + + --memory SIZE EVAL_UNDER_9P_VIRTIO_MEMORY 2G + Guest RAM (qemu -m syntax). Overlay writes live in guest RAM, so + a suite that writes a lot outside the mount needs headroom. + + --cpus N EVAL_UNDER_9P_VIRTIO_CPUS nproc + Guest vCPUs. + + --msize BYTES EVAL_UNDER_9P_VIRTIO_MSIZE (unset) + Client request size. Unset = kernel default (128 KiB since 5.15; + virtio transport caps at 512000). + + --cache MODE EVAL_UNDER_9P_VIRTIO_CACHE (unset) + v9fs cache mode. Unset = kernel default (no caching, every + kernel era). `loose` is the classic vagrant stale-read foot-gun. + + --writeout immediate EVAL_UNDER_9P_VIRTIO_WRITEOUT (unset) + Pass writeout=immediate to the virtfs server -- what + vagrant-libvirt sets (wrpolicy="immediate"). Unset = QEMU default. + + --run-as-root EVAL_UNDER_9P_VIRTIO_RUN_AS_ROOT (unset) + Keep the wrapped command running as root inside the guest + (default drops to the invoking user with runuser). The 9p analog + of eval-under-nfs's --no-root-squash; needed by pjdfstest and + stress-ng's chown/mknod stressors. Under mapped-xattr their + privileged ops "succeed" into xattr metadata -- that is the + measurement, see GOTCHAS.md. + + --share-rw PATH EVAL_UNDER_9P_VIRTIO_SHARE_RW (none) + Host directory to share read-write into the guest at the same + path (vng --rwdir). Repeatable; env var is space-separated. + CAVEAT (measured): the share is not uid-faithful -- writes + arrive on the host as the daemon's identity (root when this + script runs under sudo), and a non-root guest process can fail + with EACCES inside a directory it just created. Fine for + root-only workflows; for suite outputs use --copy-out instead. + + --copy-out PATH EVAL_UNDER_9P_VIRTIO_COPY_OUT (none) + Guest path (typically under one of vng's overlays, whose writes + are otherwise discarded with the guest) to copy back to the same + path on the host after the command finishes. Travels through the + 9p export itself, so it needs no extra share; ownership on the + host is normalized to the invoking user. Repeatable; env var is + space-separated. CI uses it for git's t/test-results. + + --vm-timeout S EVAL_UNDER_9P_VIRTIO_VM_TIMEOUT 0 (off) + Host-side timeout around the whole guest. The per-target timeout + CI wraps around CMD runs *inside* the guest and cannot catch a + hung boot; this one can. Exit 124 = the VM was killed. + + --allow-tcg EVAL_UNDER_9P_VIRTIO_ALLOW_TCG (unset) + Proceed without /dev/kvm (pure emulation, an order of magnitude + slower). Default is to fail fast with a clear message instead of + mutely blowing every suite budget. + + --log FILE EVAL_UNDER_9P_VIRTIO_LOG /var/log/eval-under-9p-virtio.log + Host file receiving a copy of all guest console/suite output + (falls back to a mktemp path when unwritable, e.g. unprivileged + local runs). What bin/ci/dump-failure-logs.sh and the CI artifact + pick up after teardown. + + --debug-boot EVAL_UNDER_9P_VIRTIO_DEBUG_BOOT (unset) + Add vng --show-boot-console --verbose: boot messages are + suppressed by default, so a boot-time failure is otherwise + silent. + + -h, --help + Print this help. + +Internal (visible so a stuck run can be debugged by hand): + + --guest-stage2 ... + The half of this script that runs inside the guest, as guest + root: modprobe + mount + usability probe + user drop + exec CMD. + The host half passes every argument shell-quoted through vng's + single command-string channel. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = (a guest path here) + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + PATH = the host-side PATH, passed through + +Exit status: the wrapped command's, carried out of the guest by vng. +Caveats: 124 with --vm-timeout means the VM itself was killed; 255 is +also vng's own "guest died before reporting" sentinel, so a command +exiting 255 is indistinguishable from a crash. No network inside the +guest. Teardown removes the host backing dir (unless --keep). +EOF +} + +log() { printf '\nI: %s\n' "$*"; } + +# --------------------------------------------------------------------------- +# Guest half. Runs first so none of the host-side setup below executes in +# the guest. Everything here runs as guest root (vng script default). +# --------------------------------------------------------------------------- +if [ "${1:-}" = "--guest-stage2" ]; then + shift + G_TAG="" G_MNT="" G_MOPTS="" G_PATH="$PATH" G_DROP_TO="" G_SET_HOME=0 G_SHELL=0 + G_COPY_OUT=() + while [ $# -gt 0 ]; do + case "$1" in + --tag) G_TAG="$2"; shift 2 ;; + --mount-point) G_MNT="$2"; shift 2 ;; + --mount-opts) G_MOPTS="$2"; shift 2 ;; + --path) G_PATH="$2"; shift 2 ;; + --drop-to) G_DROP_TO="$2"; shift 2 ;; + --set-home) G_SET_HOME=1; shift ;; + --shell) G_SHELL=1; shift ;; + --copy-out) G_COPY_OUT+=("$2"); shift 2 ;; + --) shift; break ;; + *) echo "guest-stage2: unknown arg: $1" >&2; exit 2 ;; + esac + done + if [ -z "$G_TAG" ] || [ -z "$G_MNT" ] || [ -z "$G_MOPTS" ]; then + echo "guest-stage2: --tag/--mount-point/--mount-opts required" >&2 + exit 2 + fi + + guest_dmesg_tail() { + echo "=== guest dmesg (9p/virtio, last 30) ===" >&2 + dmesg 2>/dev/null | grep -iE '9p|virtio' | tail -30 >&2 || true + } + + # Module load: a pinned mainline kernel has these preloaded by virtme's + # MODALIASES, the host kernel usually not. Best-effort; the mount is + # the real test. + modprobe -qa 9p 9pnet_virtio 2>/dev/null || true + + mkdir -p "$G_MNT" || { echo "guest-stage2: cannot mkdir $G_MNT (read-only rootfs path? see --mount-point in --help)" >&2; exit 7; } + if ! mount -t 9p -o "$G_MOPTS" "$G_TAG" "$G_MNT"; then + echo "ERROR: mount -t 9p -o $G_MOPTS $G_TAG $G_MNT failed in the guest" >&2 + echo " (a missing mount tag usually means the -virtfs device is not visible)" >&2 + guest_dmesg_tail + exit 7 + fi + + # Probe with a real create + write + read-back before handing over. + probe="$G_MNT/.eval-under-ready.$$" + if ! dd if=/dev/urandom of="$probe" bs=4k count=1 2>/dev/null \ + || ! dd if="$probe" of=/dev/null bs=4k count=1 2>/dev/null; then + echo "ERROR: 9p mounted at $G_MNT but a create+write+read probe failed" >&2 + guest_dmesg_tail + exit 6 + fi + rm -f "$probe" 2>/dev/null || true + + echo "I: guest kernel $(uname -r)" + grep -F " $G_MNT " /proc/mounts | sed 's/^/I: /' || true + + if [ -n "$G_DROP_TO" ]; then + chown "$G_DROP_TO" "$G_MNT" 2>/dev/null || true + fi + + RUN_HOME="${HOME:-/root}" + if [ "$G_SET_HOME" = 1 ]; then + RUN_HOME="$G_MNT/home" + if [ -n "$G_DROP_TO" ]; then + runuser -u "$G_DROP_TO" -- mkdir -p "$RUN_HOME" + else + mkdir -p "$RUN_HOME" + fi + fi + + if [ "$G_SHELL" = 1 ]; then + echo "I: interactive guest shell; the mount is at $G_MNT (exit to end the VM)" + export HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" + if [ -n "$G_DROP_TO" ]; then + exec runuser -u "$G_DROP_TO" -- bash -i + fi + exec bash -i + fi + + echo "I: running in guest${G_DROP_TO:+ (as $G_DROP_TO)}: $*" + rc=0 + if [ -n "$G_DROP_TO" ]; then + runuser -u "$G_DROP_TO" -- \ + env HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" \ + "$@" || rc=$? + else + env HOME="$RUN_HOME" TMPDIR="$G_MNT" DATALAD_TESTS_TEMP_DIR="$G_MNT" PATH="$G_PATH" \ + "$@" || rc=$? + fi + # Ferry requested guest paths (typically overlay content that would + # otherwise vanish with the guest) out through the 9p export, as + # guest root, whatever the command's outcome -- failed runs are when + # the results matter most. + for p in ${G_COPY_OUT[@]+"${G_COPY_OUT[@]}"}; do + if [ -e "$p" ]; then + dest="$G_MNT/.eval-under-copyout$(dirname "$p")" + if mkdir -p "$dest" && cp -a "$p" "$dest/"; then + echo "I: copied out $p" + else + echo "W: copy-out of $p failed" >&2 + fi + else + echo "I: copy-out path $p does not exist in the guest; skipping" + fi + done + if [ "$rc" -ne 0 ]; then + guest_dmesg_tail + fi + exit "$rc" +fi + +# --------------------------------------------------------------------------- +# Host half. +# --------------------------------------------------------------------------- +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --shell) SHELL_MODE=1; shift ;; + --security-model) SECMODEL="$2"; shift 2 ;; + --kernel) KERNEL="$2"; shift 2 ;; + --memory) MEMORY="$2"; shift 2 ;; + --cpus) CPUS="$2"; shift 2 ;; + --msize) MSIZE="$2"; shift 2 ;; + --cache) CACHE="$2"; shift 2 ;; + --writeout) WRITEOUT="$2"; shift 2 ;; + --run-as-root) RUN_AS_ROOT=1; shift ;; + --share-rw) SHARE_RW+=("$2"); shift 2 ;; + --copy-out) COPY_OUT+=("$2"); shift 2 ;; + --vm-timeout) VM_TIMEOUT="$2"; shift 2 ;; + --allow-tcg) ALLOW_TCG=1; shift ;; + --log) LOG="$2"; shift 2 ;; + --debug-boot) DEBUG_BOOT=1; shift ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [ "$SHELL_MODE" = 0 ] && [ $# -eq 0 ]; then + echo "no command given" >&2; usage >&2; exit 2 +fi + +case "$SECMODEL" in + mapped|mapped-xattr|mapped-file|passthrough|none) ;; + *) echo "unknown --security-model: $SECMODEL (mapped-xattr|mapped-file|passthrough|none)" >&2; exit 2 ;; +esac + +# The user the wrapped command should run as inside the guest (the guest +# sees the host's /etc/passwd through the shared rootfs). +if [ -n "${SUDO_USER:-}" ]; then + INVOKER_USER="$SUDO_USER"; INVOKER_UID="${SUDO_UID:-0}" + INVOKER_GID="${SUDO_GID:-$INVOKER_UID}" +else + INVOKER_USER="$(id -un)"; INVOKER_UID="$(id -u)" + INVOKER_GID="$(id -g)" +fi +[ "$INVOKER_UID" = 0 ] && RUN_AS_ROOT=1 # dropping root to root is a no-op + +ensure_virtio_installed() { + command -v vng >/dev/null 2>&1 || { + echo "ERROR: virtme-ng (vng) not installed." >&2 + echo " Install with: apt install virtme-ng (Ubuntu 24.04+)" >&2 + echo " or: pipx install virtme-ng (22.04 and other distros)" >&2 + exit 3 + } + command -v qemu-system-x86_64 >/dev/null 2>&1 || { + echo "ERROR: qemu-system-x86_64 not installed." >&2 + echo " Install with: apt install qemu-system-x86" >&2 + exit 3 + } + if [ "$KERNEL" != host ] && ! command -v busybox >/dev/null 2>&1; then + echo "ERROR: a pinned --kernel needs an initramfs, which vng builds with busybox." >&2 + echo " Install with: apt install busybox-static" >&2 + exit 3 + fi + if [ "$KERNEL" != host ] && ! command -v zstd >/dev/null 2>&1; then + echo "ERROR: mainline kernel debs ship .ko.zst modules; vng needs zstd for them" >&2 + echo " (the failure mode without it is an opaque guest death, exit 255)." >&2 + echo " Install with: apt install zstd" >&2 + exit 3 + fi + if [ "$KERNEL" = host ]; then + local vmlinuz + vmlinuz="/boot/vmlinuz-$(uname -r)" + if [ -e "$vmlinuz" ] && [ ! -r "$vmlinuz" ]; then + echo "ERROR: $vmlinuz is not readable (Ubuntu ships it 0600)." >&2 + echo " Run under sudo, chmod +r it, or use --kernel v instead." >&2 + exit 3 + fi + fi + if [ ! -e /dev/kvm ]; then + if [ "$ALLOW_TCG" != 0 ]; then + echo "W: no /dev/kvm -- continuing under TCG emulation (SLOW)" >&2 + else + echo "ERROR: no /dev/kvm. A TCG-emulated guest is an order of magnitude" >&2 + echo " slower and blows every suite budget; pass --allow-tcg to insist." >&2 + exit 3 + fi + fi + if ! command -v virtiofsd >/dev/null 2>&1 \ + && [ ! -x /usr/lib/virtiofsd ] && [ ! -x /usr/libexec/virtiofsd ] \ + && [ ! -x /usr/lib/qemu/virtiofsd ]; then + echo "W: no virtiofsd found; vng will fall back to a 9p rootfs (slower boot," >&2 + echo " and the root filesystem shares the transport under test)." >&2 + fi + # The guest boots this host's own rootfs, so guest-side plumbing + # depends on host packages: virtme-init needs udevd to surface the + # virtio-serial script ports (without it the guest powers off with + # "cannot find script I/O ports" and vng reports its 255 sentinel). + # Present on any normal Ubuntu install; missing in minimal containers. + if ! command -v udevadm >/dev/null 2>&1; then + echo "W: no udev on this host -- the guest will likely fail with" >&2 + echo " 'cannot find script I/O ports'. Install with: apt install udev" >&2 + fi +} + +# Host-side backing dir that QEMU exports. Its path is embedded in a +# comma-separated -virtfs value and in a space-joined qemu-opts string, +# so refuse the characters that would silently corrupt those. +ORIG="$(mktemp -d "${TMPDIR:-/tmp}/eval-under-9p-XXXXX.orig")" +case "$ORIG" in + *[,\ ]*) echo "ERROR: backing dir path '$ORIG' contains a comma or space (set TMPDIR elsewhere)" >&2; exit 2 ;; +esac + +if [ -z "$LOG" ]; then + LOG=/var/log/eval-under-9p-virtio.log + if ! touch "$LOG" 2>/dev/null; then + LOG="$(mktemp "${TMPDIR:-/tmp}/eval-under-9p-virtio-XXXXX.log")" + fi +fi + +MOPTS="trans=virtio,version=9p2000.L" +[ -n "$MSIZE" ] && MOPTS="$MOPTS,msize=$MSIZE" +[ -n "$CACHE" ] && MOPTS="$MOPTS,cache=$CACHE" + +VIRTFS="local,path=$ORIG,mount_tag=$TAG,security_model=$SECMODEL,id=$TAG" +[ -n "$WRITEOUT" ] && VIRTFS="$VIRTFS,writeout=$WRITEOUT" + +build_vng_cmd() { + VNG_CMD=(vng) + [ "$KERNEL" != host ] && VNG_CMD+=(--run "$KERNEL") + # microvm has no PCI bus the stock guest kernels can enumerate; the + # -virtfs device below is virtio-9p-pci, so force a PCI-ful machine. + VNG_CMD+=(--disable-microvm --memory "$MEMORY" --cpus "$CPUS") + local d + for d in ${SHARE_RW[@]+"${SHARE_RW[@]}"}; do + VNG_CMD+=(--rwdir "$d") + done + # =-joined: the value starts with a dash and argparse would otherwise + # read it as the next option. + VNG_CMD+=("--qemu-opts=-virtfs $VIRTFS") + if [ "$DEBUG_BOOT" != 0 ]; then + # vng 1.22 has no separate boot-console flag; --verbose shows the + # boot console and the underlying virtme-run line. + VNG_CMD+=(--verbose) + fi +} + +# The command travels through exactly two shell layers, so quoting is +# OUR job, done in two matching passes: +# +# layer 1: vng embeds the joined words after `--` verbatim inside +# double quotes in a `virtme-run ... --script-sh "..."` +# line and runs it via `sh -c` (check_call(shell=True)) -- +# a POSIX double-quote context, which consumes one level of +# \\ \" \$ \` escapes; +# layer 2: virtme-run base64s the surviving string through the kernel +# cmdline and the guest executes it as a *bash* script. +# +# So: printf %q makes each argument safe for the guest bash (layer 2), +# and dq_escape pre-compensates the double-quote context (layer 1) so +# the %q text arrives in the guest byte-identical. Without the second +# pass, a wrapped `bash -c 'cd "$HOME" && ...'` would have $HOME +# expanded one layer early, as guest root. +dq_escape() { + local s="$1" + s=${s//\\/\\\\} + s=${s//\"/\\\"} + s=${s//\$/\\\$} + s=${s//\`/\\\`} + printf '%s' "$s" +} + +build_stage2_string() { + local args=("$SELF" --guest-stage2 + --tag "$TAG" --mount-point "$MNT" --mount-opts "$MOPTS" + --path "$PATH") + [ "$SET_HOME" = 1 ] && args+=(--set-home) + [ "$RUN_AS_ROOT" = 0 ] && args+=(--drop-to "$INVOKER_USER") + [ "$SHELL_MODE" = 1 ] && args+=(--shell) + local c + for c in ${COPY_OUT[@]+"${COPY_OUT[@]}"}; do + args+=(--copy-out "$c") + done + args+=(--) + local a + STAGE2_STR="" + for a in "${args[@]}" "$@"; do + STAGE2_STR+="$(dq_escape "$(printf '%q' "$a")") " + done +} + +# Invoked only via the EXIT trap; shellcheck's reachability analysis +# cannot see that and calls the body unreachable (same annotation as +# matrix.sh carries for its trap/return fallback). +# shellcheck disable=SC2317 +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving backing dir + log (exit=$rc)" + echo "I: backing dir: $ORIG (mapped-xattr metadata lives in user.virtfs.* xattrs)" + echo "I: guest log: $LOG" + echo "I: relaunch interactively with: $0 --shell ${SECMODEL:+--security-model $SECMODEL}" + return "$rc" + fi + rm -rf "$ORIG" 2>/dev/null || true + # $LOG is deliberately kept: it is what bin/ci/dump-failure-logs.sh + # and the CI artifact read after this process is gone. + return "$rc" +} +trap teardown EXIT + +ensure_virtio_installed +build_vng_cmd +build_stage2_string "$@" + +log "guest: kernel=$KERNEL mem=$MEMORY cpus=$CPUS log=$LOG" +if [ "${#SHARE_RW[@]}" -gt 0 ]; then + log "rw dirs shared into guest: ${SHARE_RW[*]}" +fi +log "virtfs: $VIRTFS" +log "mount (in guest): -t 9p -o $MOPTS $TAG $MNT" + +RUNNER=("${VNG_CMD[@]}" -- "$STAGE2_STR") +if [ "$VM_TIMEOUT" != 0 ]; then + RUNNER=(timeout --kill-after=30 "$VM_TIMEOUT" "${RUNNER[@]}") +fi + +rc=0 +if [ "$SHELL_MODE" = 1 ]; then + # Interactive: no tee (it would steal the terminal). + "${RUNNER[@]}" || rc=$? +else + "${RUNNER[@]}" 2>&1 | tee -a "$LOG" || true + rc="${PIPESTATUS[0]}" +fi + +# Retrieve --copy-out payloads from the export's backing dir onto their +# real host paths, normalizing ownership: under mapped-xattr the +# backing files carry the QEMU process's uid and 0600/0700 modes, which +# a later artifact-upload step could not even read. +for p in ${COPY_OUT[@]+"${COPY_OUT[@]}"}; do + src="$ORIG/.eval-under-copyout$p" + if [ -e "$src" ]; then + mkdir -p "$(dirname "$p")" + cp -r --no-preserve=mode,ownership "$src" "$(dirname "$p")/" + if [ "$(id -u)" -eq 0 ]; then + chown -R "$INVOKER_UID:$INVOKER_GID" "$p" + fi + log "copied back $p" + else + echo "W: no copy-out payload for $p (guest never wrote it?)" >&2 + fi +done + +if [ "$rc" -eq 124 ] && [ "$VM_TIMEOUT" != 0 ]; then + echo "W: exit 124 -- either the wrapped command hit its own timeout inside the" >&2 + echo " guest (a dmesg tail above means the guest was alive to report it), or" >&2 + echo " the VM exceeded --vm-timeout ${VM_TIMEOUT}s and was killed from outside." >&2 +elif [ "$rc" -eq 255 ]; then + echo "W: exit 255 -- either the wrapped command exited 255, or the guest died" >&2 + echo " before reporting (vng's sentinel). Re-run with --debug-boot to see boot." >&2 +fi +exit "$rc" diff --git a/drafts/9p-backend-plan.md b/drafts/9p-backend-plan.md new file mode 100644 index 0000000..683f3f6 --- /dev/null +++ b/drafts/9p-backend-plan.md @@ -0,0 +1,142 @@ +# 9p backends: design record + +Status: **implemented** (both backends, CI rows `9p-tcp / n/a` and +`9p-virtio / mapped`). This file is the decision record from the design +review; the operational truth lives where it belongs -- settings and +semantics in [GOTCHAS.md](../GOTCHAS.md), knobs in each script's +`--help`, deferred rows in GOTCHAS "Not yet covered". Delete this file +once those deferred rows have either landed or been rejected. + +## Why 9p + +9p is the filesystem people get, usually without choosing it, when a +directory is shared across a VM boundary: Vagrant + QEMU/libvirt +`type: "9p"` synced folders (QEMU's virtfs server + the kernel v9fs +client over virtio), WSL2's `/mnt/c` (Microsoft's 9p server), Chrome OS +crostini, pre-virtiofs kata. Its quirk classes -- cache staleness, +server-faked or whole-file locking, absorbed chown/mknod under mapped +security models, msize cliffs, no `O_TMPFILE`, no remote change +notification -- are disjoint from every existing row, and our own +Vagrantfile dodges them ("rsync ... avoids permission surprises") +rather than measuring them. + +## The two mechanisms, and why both exist + +- **`bin/eval-under-9p-tcp`** -- diod (9P2000.L) on localhost TCP, + kernel client mount on the same host. Architecturally the NFS + backend's sibling; no VM, no KVM, server runs unprivileged. The + everyday local-debugging 9p, and the cheap CI row. +- **`bin/eval-under-9p-virtio`** -- QEMU `-virtfs local,...` into a + virtme-ng guest booted from the host's own rootfs, suite runs inside + the guest. The only way to exercise the *actual* server vagrant users + hit; virtme-ng is what makes it fit the harness (host-installed + targets exist in the guest unchanged, exit status and stdio are + plumbed out over virtio-serial). + +Two scripts rather than one `--transport` flag: they share the string +"9p" and the env-trio contract and nothing else -- different daemon vs +hypervisor, privilege model, teardown, diagnostics (host dmesg vs a +guest that no longer exists). One mechanism per script is the house +pattern (loop parametrizes filesystems, beegfs parametrizes versions; +neither multiplexes mechanisms). + +## Decisions of record + +Two independent design reviews (one systems-mechanics, one +harness-fit) converged on the shape that got implemented. The calls, +and what settled them: + +1. **Both transports landed together, tcp as the low-risk row.** The + interesting client semantics are identical across both (same v9fs); + the servers differ instructively (diod: whole-file `flock`; QEMU: + TLOCK-always-succeeds). The virtio row carries the CI unknowns, so + the tcp row guarantees the matrix gains a working 9p row even if + virtio needs iteration. +2. **Backends never read `.github/matrix.yaml`.** That line is what + keeps `bin/eval-under-*` first-class local tools. The pinned guest + kernel (`refs: 9p-kernel`, mainline build fetched by `vng --run`) + is handed to the backend *by `bin/ci/run-under.sh`*, exactly like + `--size` and `--no-root-squash`; the script's own default is the + host kernel -- zero downloads, works offline. +3. **Guest kernel pinned in CI, host kernel locally.** The v9fs client + *is* the kernel and moved substantially in 6.4 (cache-mode rework) + and 6.8 (netfs buffered writes); unpinned, a newly-red cell cannot + distinguish filesystem regression from client drift. This is a + deliberate departure from the "distro defaults, not pinned" + environment philosophy (NFS protocol version is deliberately + unpinned) -- argued, not smuggled: for 9p the client is the thing + under test's other half. +4. **`--disable-microvm`.** vng's microvm machine has no PCI bus the + stock Ubuntu/mainline guest kernels can enumerate, and `-virtfs` is + virtio-9p-pci -- on the exact recommended path (24.04 + KVM + + virtiofsd) the mount tag would silently never appear. +5. **Root only where root is needed.** tcp: diod runs unprivileged in + single-user mode, sudo is for mount/umount; virtio: nothing on the + host needs root at all (mounting happens inside the guest, QEMU and + vng run unprivileged). Guest scripts run as root, so the wrapped + command is *dropped* to the invoking user via `runuser` -- vng's + default would otherwise measure root-in-a-VM. `--run-as-root` (both + scripts) is the 9p analog of NFS `--no-root-squash`, wired to + `target_needs_root()` in run-under.sh; without it the pjdfstest + cell would be a contentless setup-failure red. +6. **Guest mountpoint defaults to `/tmp/eval-under-9p`.** vng's rootfs + is read-only outside its overlay set; `/mnt` would EROFS. Overlay + writes evaporate with the guest, and vng's `--rwdir` shares turned + out not to be uid-faithful (measured; see GOTCHAS) -- so git's + `t/test-results/**` reaches the runner via the backend's + `--copy-out`, which ferries guest paths back through the 9p export + itself with ownership normalized. +7. **Host-side `--vm-timeout` around the whole guest.** The per-target + `timeout` that run-under.sh wraps around the suite travels *into* + the guest and cannot catch a boot or mount hang; CI passes + target-timeout + 300s. +8. **Diagnostics survive the guest.** All guest output is teed to + `/var/log/eval-under-9p-virtio.log` (in the artifact list), stage2 + appends the guest dmesg tail on failure, and the backend + disambiguates vng's exit sentinels (124 = VM killed by + `--vm-timeout`; 255 = crash-or-genuine-255). The backing dir is + teardown-deleted before `dump-failure-logs.sh` runs, so nothing + diagnostic may live only there. +9. **Per-row `runs-on` in matrix.yaml** (default ubuntu-22.04; 9p rows + ubuntu-24.04, where virtme-ng/virtiofsd are packaged). Costed + knowingly: the backend tuple grew a fourth field across matrix.sh / + matrix-json.sh / gen-readme-matrix.sh, and rows on different images + differ in host kernel and tool versions -- acceptable here because + the virtio row's client kernel is pinned anyway, recorded so the + next person prices it too. +10. **No separate probe workflow.** The durable assertions live in + `install-backend.sh` (modprobe + /proc/filesystems for tcp; udev + KVM rule, `vng --version`, and a pre-warm boot of the pinned + kernel for virtio -- which doubles as the boot smoke and moves the + ~180 MB mainline download outside the suite's timeout). The PR's + own matrix run is the integration probe. + +## Facts the reviews settled (so nobody re-derives them) + +- 9p client modules (`9p`, `9pnet`, `9pnet_fd`, `9pnet_virtio`) ship in + the kernel's **base `linux-modules`** package on Ubuntu generic, + virtual and azure flavours alike -- `linux-modules-extra` is NOT + needed anywhere in this design. (`9pnet_tcp` does not exist; + `trans=tcp` lives in `9pnet_fd`.) +- Standard GitHub-hosted Linux runners have KVM (officially since + 2024); the udev rule in `install_9p_virtio()` is the documented + enablement for non-root use. +- `vng --run` wants a tag-shaped version (`v6.8`), downloads the Ubuntu + mainline image+modules (~180 MB, cached in `~/.cache/virtme-ng`), + and those builds carry the 9p modules; vng propagates the guest + command's exit code over a dedicated channel, with 255 as its + crash sentinel. +- vagrant-libvirt's *default* accessmode is `passthrough` under an + unprivileged QEMU -- neither of the first two virtio rows; recorded + with a local repro recipe in GOTCHAS "Not yet covered". +- diod caps msize at 64 KiB and implements Tlock as whole-file + `flock()`; QEMU's virtfs answers every TLOCK with success. Same + client, two instructively different servers. + +## Deferred (tracked in GOTCHAS "Not yet covered") + +`9p-virtio / passthrough` (root QEMU) and the true vagrant default +(passthrough, unprivileged QEMU); `cache=loose` and small-msize +variants; an `eval-under-virtiofs` sibling backend as the "is this 9p +or any VM-shared fs?" control row. All are flag-reachable locally +today; each becomes one matrix line when promoted. diff --git a/provision/setup.sh b/provision/setup.sh index d6f7ab6..9f06d9e 100755 --- a/provision/setup.sh +++ b/provision/setup.sh @@ -43,6 +43,18 @@ apt-get install -y --no-install-recommends \ nfs-kernel-server \ dosfstools xfsprogs btrfs-progs +log "eval-under-9p-tcp / eval-under-9p-virtio dependencies" +# diod: 9P2000.L server for eval-under-9p-tcp. +# virtme-ng: boots the eval-under-9p-virtio guest (vng); the VM's +# nested KVM (Vagrantfile: lv.nested) accelerates it. +# qemu-system-x86: virtme-ng only Recommends it; --no-install-recommends. +# virtiofsd: vng rootfs transport (falls back to 9p without it). +# busybox-static: vng initramfs for pinned --kernel downloads. +# zstd: mainline kernel debs ship .ko.zst modules. +# 9p client modules ship in the kernel's base linux-modules package. +apt-get install -y --no-install-recommends \ + diod virtme-ng qemu-system-x86 qemu-utils virtiofsd busybox-static zstd + log "eval-under test-target dependencies" # stress-ng: bin/ci/target-stress-ng.sh (apt is the whole install). # autoconf/automake/libtool: building pjdfstest from its pinned tag.