Add filesystem probing and sshfs backend for eval-under - #5
Open
yarikoptic-gitmate wants to merge 19 commits into
Open
yarikoptic-gitmate wants to merge 19 commits into
yarikoptic-gitmate wants to merge 19 commits into
Conversation
Two scripts and a workflow that answer, empirically, which further filesystems this harness could grow a backend for. bin/ci/fs-capabilities.sh reports what a mounted filesystem actually supports, in the dimensions git-annex and DataLad are known to trip over: symlinks, hardlinks, fifos, exec/permission bits, POSIX record locks, flock, SQLite in WAL mode, xattrs, case sensitivity, timestamp granularity, and whether link() correctly refuses to clobber an existing name. Each check maps to a reported failure rather than to a generic conformance checklist -- sqlite-wal=no is the direct cause of "SQLite3 returned ErrorIO" on Lustre and WSL, and link-eexist=no is what makes annex.pidlock unsafe on Lustre. bin/ci/probe-backend.sh tries to stand up one candidate filesystem on a stock runner, runs the capability probe on it, and tears it down. The exit status is the finding: green means a bin/eval-under-<name> backend is possible on GitHub-hosted infrastructure, red means it is not, with the reason (no kernel module / no package / needs a cluster / needs a VM) in the log. Deliberately runs without `set -e` -- a probe that dies on its first failed command reports nothing. Candidates cover the cluster filesystems with a single-node mode (gfs2, ocfs2), loop filesystems the loop backend does not reach yet (f2fs, exfat, ntfs3, udf, nilfs2, bcachefs, zfs), the network/FUSE shapes users actually hit (glusterfs, cephfs, cifs, sshfs, s3 via rclone), the crypto layers (gocryptfs, encfs, ecryptfs), overlayfs, and Lustre's client modules. Validated locally where this container allows it: overlay and gocryptfs bootstrap and probe clean, gfs2 correctly reports the missing kernel module. The loop-backed and network candidates get their first real signal from the workflow.
First run answered the main question -- gfs2 and ocfs2 both come up on a stock runner in under 70s, cifs reproduces the SQLite failure mode in 23s -- but three cells failed without saying why, because a 50-line dmesg step buried the probe's own output past any reasonable log tail. - Drop the dmesg step and end every probe with a single machine-readable SUMMARY| line carrying verdict, elapsed time, note and the full capability profile, so `tail -3` of a job log is the whole finding. - glusterfs: print glusterd status and volume info, and register the brick under the node's own hostname. Gluster stores the brick host in the volfile and requires it to be a peer, which 127.0.0.1 need not be. - ecryptfs: echo what ecryptfs-add-passphrase actually printed, since the signature is scraped from it and an empty scrape is indistinguishable from an unsupported filesystem. - lustre-client: list the module packages the Whamcloud repo publishes. The first run showed the ubuntu2204 repo exists but has no lustre-client-modules-dkms; naming which kernels it *does* ship for turns that into a usable answer. - Add an openafs candidate: the other out-of-tree client in this space, and unlike Lustre it is packaged by Ubuntu, so DKMS builds it against the runner's kernel. - fs-capabilities: report chown as n/a when not running as root. chown(2) is restricted regardless of filesystem, so a "no" there said nothing.
The probe showed both cluster filesystems come up on a stock runner, so teach the existing loop backend to make them rather than adding two near-duplicate backends. Both are ordinary block-device filesystems once their single-node lock mode is selected at mkfs time: gfs2 -O -p lock_nolock -j 1 -J 8 ocfs2 -F -M local -N 1 -b 4K -C 8K --fs-features=local -q so the loop backend grows a per-filesystem mkfs option table (empty for everything it already supported -- the deliberate "distro defaults" choice GOTCHAS.md documents stays intact), an --mkfs-opts override, and a per-filesystem minimum image size. The last one is not cosmetic: both reserve journal space and neither fits in the 100MB default, so the size is raised to their 256MB floor unless an explicit --size says otherwise, in which case a too-small value is an error rather than a silent bump. GFS2 also gets lockproto=lock_nolock at mount time. install-backend.sh learns the matching packages (gfs2-utils, ocfs2-tools, f2fs-tools, exfatprogs, nilfs-tools, bcachefs-tools) and, for the two cluster filesystems, checks the kernel module is there -- both live in linux-modules-extra, so their absence would otherwise surface as "unknown filesystem type" at mount time. Both mkfs invocations are validated locally (this container has the tools but not the modules, so mkfs succeeds and mount reports ENODEV); a new backend-smoke job runs the real `eval-under loop --fs <fs>` path on a runner that does have them, with ext4 as the control. Also adds a roll-up job: each probe uploads its SUMMARY line and one final job renders them as a single table, so a run can be read without opening two dozen job logs.
The survey behind the loop-backend and probe changes: which filesystems have actually broken git-annex or DataLad for real users (from the trackers, with links), which of those can be stood up on a stock runner (measured by the probe workflow, not guessed), and what follows. Four mechanisms account for nearly every report -- SQLite unable to lock its WAL, absent POSIX record locks, link() that does not refuse an existing name, and no symlinks/exec bit -- which is why bin/ci/fs-capabilities.sh checks those specifically. Answers the two filesystems that prompted it. "GFS" is three different filesystems: GFS2 is nearly free (single-node lock_nolock on a loop device), GlusterFS bootstraps from Ubuntu packages, and GPFS is not possible in public CI -- IBM's free Developer Edition is real but sits behind an IBM ID with no redistributable repo, so it needs a self-hosted runner. Lustre is walled off twice on a GitHub runner: Whamcloud ships client modules prebuilt for 5.15 generic kernels while the runners are on 6.8-azure with no DKMS package to fall back on, and the server needs either a patched kernel or a ZFS OSD. The recommended path is Lustre's own llmount.sh single-node loopback config inside the Vagrant VM first, with a KVM-in-CI backend only if the answer proves interesting. GOTCHAS.md records the new gfs2/ocfs2 mkfs and mount knobs, and says plainly what a single-node cluster filesystem does and does not measure. README points at the survey and lists the new files.
The floor introduced with gfs2/ocfs2 support errored out when a size was passed explicitly, on the theory that an explicit request deserves an explicit refusal. That is wrong here: bin/ci/run-under.sh always passes --size, taken from target_loop_size_mb(), which is chosen per *target* and knows nothing about a filesystem's journal overhead. So promoting gfs2 or ocfs2 to a matrix row would have failed immediately with "--size 100 is below the 256MB minimum" -- on the very path the change was written to enable. Raise unconditionally and log it. No caller can be expected to know every filesystem's minimum, and the harness's job is to produce the smallest working filesystem of the requested type, not to litigate the request.
…easured The ceph demo container pulls about a gigabyte and then may never reach a responsive cluster; on the last run it held the whole matrix open until its 25-minute job timeout, which also blocked the roll-up that makes the run readable. Cap the wait at 4 minutes and dump the container's own log on the way out: "not on a runner" is a verdict, and it should arrive in four minutes rather than twenty-five. FILESYSTEMS.md: separate the Lustre facts that were measured from the ones that are cited. Measured -- the runner kernels (6.8.0-1064-azure on 22.04, 6.17.0-1022-azure on 24.04), no lustre-client-modules-dkms in latest-release/ubuntu2204/client, no ubuntu2404 client repo at all, and /dev/kvm present with qemu-system-x86_64 not installed. Cited -- that Whamcloud's index carries per-kernel module packages built for 5.15 generics. The earlier text asserted one runner kernel for both images, which was wrong.
Capping the poll loop at 4 minutes did not cap anything: with no reachable mon, `ceph -s` does not fail, it blocks for minutes inside its own retry loop, so 48 iterations of "sleep 5 then ask again" ran for as long as the job would let it. Two runs in a row ended with cephfs holding the roll-up open while every other candidate had long since reported. Wrap each ceph client call in `timeout` (and pass --connect-timeout), and bound the `docker run` that pulls the ~1GB demo image. The loop is now genuinely time-bounded: roughly two minutes of polling on top of a bounded pull, then a verdict either way.
Twenty-four candidates, one runner, real numbers. Sixteen bootstrap; four do not; and three of the measurements change what a row would be for, so they are called out rather than left in the table: - CIFS is the only candidate that fails both sqlite-wal and sqlite-delete-mode -- the exact mechanism behind the Samba and Lustre reports, reproduced in 26 seconds on a localhost share. - sshfs resolves five rapid creates into one distinct mtime where every other filesystem here gives five. Nothing in the matrix currently tests a coarse clock against git's racy-timestamp handling. - ntfs3 is *not* a crippled filesystem: symlinks, mode bits and xattrs all work, so it would test the in-kernel NTFS driver rather than being a second vfat. Also recorded as a negative: every single-node filesystem reachable from a runner passes link-eexist, the POSIX rule Lustre is reported to break. The check stays, but that behaviour cannot be reproduced without Lustre. The four failures now carry their measured reason -- openafs and lustre both fail to build out-of-tree modules against the runners' Azure kernels, cephfs never converges inside a bounded budget, and ecryptfs refuses the mount options the probe passes (a scripting gap, not a proven impossibility). Tier lists updated accordingly.
The matrix answers "is this filesystem broken?" on a schedule. Support needs the other question answered on demand: someone reports a problem on a filesystem we do not have a cell for, and we want that exact combination running now, with their mount options, without adding a badge that nobody will look at again. bin/eval-under-sshfs brings up a throwaway sshd -- its own port, host key, authorized_keys and pid file, all inside the run's scratch dir -- and sshfs-mounts a fresh backing directory back over it. Deliberately not the system sshd: that would mean appending to the user's ~/.ssh/authorized_keys and trusting teardown to remove it again. With --host it mounts a real remote instead, using the caller's ssh config, because for an sshfs report the interesting variable is often the server's SFTP implementation rather than sshfs itself. Two knobs matter enough to be flags rather than raw -o strings: --no-cache turns off the attribute cache that hides stale-stat behaviour, and --workaround rename makes sshfs emulate rename-over-existing by unlinking first -- a non-atomic rename, which is exactly what git and git-annex assume they can rely on. reproduce.yaml is the support entry point: backend, version, target, backend options and runner image are all dispatch inputs. No cron, no badge, no push trigger. It delegates to _test-under.yaml, which grows `runner` and `backend-options` inputs (both defaulted, so the twenty generated cells are unchanged). `capabilities` joins as an on-demand-only target: bin/ci/fs-capabilities.sh under any backend, seconds rather than minutes, and usually enough to explain a suite result before the suite is worth starting. It is listed in EVAL_UNDER_ONDEMAND_TARGETS rather than EVAL_UNDER_TARGETS, so gen-dispatchers.sh still generates exactly 20 cells. Validated locally: basic ops, --no-cache, --workaround rename (verified rename-over-existing still lands), clean teardown leaving no mount, no scratch dir and no stray sshd, and `run-under.sh sshfs n/a capabilities` end to end. shellcheck, YAML parse, gen-dispatchers --check and reuse lint all clean.
Adds a "Reproducing a report" section: capability profile first because it takes seconds and usually explains the failure, then the suite under the same backend, then the same thing from the Actions tab for people who would rather not have a local mount. Plus the sshfs backend in the CLI examples and the file-layout table.
Two things the first real use of the backend turned up. The port was fixed at 2222, so a second run while one was in flight died with "sshd did not start" -- which is exactly what happens when you try to reproduce something while a suite is already running, and would also bite two CI cells sharing a runner. Default is now the first free port at or above 2222; --port still pins it. The bigger one: `hardlink=yes` was measuring the wrong thing. On sshfs `ln` succeeds, and then both names report a distinct inode and nlink=1 -- the link exists on the server but is not observable as one through the mount, because SFTP has no way to express it. That is not a misconfiguration and there is no sshfs option to fix it (use_ino was FUSE2; sshfs 3.x only offers disable_hardlink, which goes the other way). It matters because git-annex's add on an adjusted unlocked branch hardlinks the file into .git/annex/objects and then verifies the link, and git's local clone does the same check. Both fail on such a filesystem while `ln` itself looks perfectly healthy -- so the old check reported a green cell for a filesystem where `git annex add` cannot work. hardlink-same-inode and hardlink-nlink measure that directly: both `no` on sshfs, both `yes` on ext4.
Measured with git-annex 10.20240129 against an ext4 baseline that passes the whole suite in 1m21s, so anything red here is the filesystem. The finding is one mechanism, not a list: SFTP cannot express that two names share an inode, so `ln` succeeds and then both names report distinct inodes and nlink=1. git-annex's add on an adjusted unlocked branch hardlinks content into .git/annex/objects and verifies the link; git's local clone does the same. Both fail. Locked repos and v10 unlocked pointer-file repos are unaffected -- it is the adjusted unlocked branch specifically, which is the variant DataLad leans on. Also recorded: the second-granularity clock (five rapid creates land on one or two distinct mtimes, against five everywhere else), and the absent fifos and unix sockets that git-annex notices at init and adapts to on its own.
…ed it The survey ranked sshfs on timestamp granularity. The real reason is better: it is the only candidate measured that breaks a git-annex operation outright. Recording how it was found is the point -- `hardlink=yes` called it healthy, and it took a real suite run to learn that `add` on an adjusted unlocked branch cannot work there. The probe now answers that in ten seconds instead.
Both full-suite runs stopped at Remote Tests / unavailable remote / removeKey and stayed there until killed. The ext4 control runs that same test in 0.02s and finishes the whole suite in 1m21s, so it is sshfs-specific. Worth writing down what it is *not*, since that is what a reporter would otherwise spend an afternoon on: the mount answers immediately, and git-annex has no open files on the mount, no sockets and no child processes -- its threads are parked in futex/epoll. That is an internal wait, not blocked filesystem I/O. Two caveats recorded with it: the test passes in seconds when run alone with -p, so it needs full-suite context to reproduce, and this was the distro's 10.20240129 rather than a daily build.
master replaced the generated per-cell dispatchers with one workflow over .github/matrix.yaml, so the three conflicts were all "this branch edited something that no longer exists in that form". - _test-under.yaml is gone; reproduce.yaml no longer calls a reusable workflow, it carries the same steps directly (minus the matrix fan-out and minus publishing -- a reproduction must never rewrite the status the README and report page show for master). - matrix.sh: took master's accessors-over-YAML version and taught the parser one new flag. A target marked `on-demand: true` lands in EVAL_UNDER_ONDEMAND_TARGETS instead of EVAL_UNDER_TARGETS, so target_known() accepts it while matrix-json.sh -- which iterates the latter -- generates no cell for it. `capabilities` moves out of the shell and into .github/matrix.yaml where the rest of the matrix now lives; the grid stays at exactly 20 cells, verified. - README: master's file-layout table, with the sshfs backend, the probe scripts, FILESYSTEMS.md and reproduce.yaml slotted in at its column widths. Also updated FILESYSTEMS.md's promotion instructions, which still told the reader to edit EVAL_UNDER_BACKENDS in matrix.sh and re-run gen-dispatchers.sh -- both gone. It is a data edit in .github/matrix.yaml plus gen-readme-matrix.sh now. Two codespell hits fixed, since master added that check after this branch forked: a "losd" link label (now zfsosd) and Samba's "browseable", which accepts "browsable" just as happily.
Opening the PR made two design mistakes visible. The probe exited non-zero for "NOT BOOTSTRAPPABLE", so lustre, openafs, cephfs and ecryptfs -- four candidates that are *permanently* unbootstrappable on a hosted runner, which is precisely the finding the probe exists to record -- pinned four red checks to the pull request. "Are this PR's checks green?" then has no answer, and the four reds say nothing a reader could act on. The verdict already lives in the roll-up table and the step summary, so the exit status was carrying no information. Genuine script errors still exit non-zero; they return before the verdict is recorded. The trigger was a bare `push`, so every branch push anywhere spawned ~25 reconnaissance jobs. Now: dispatch on demand, plus master pushes that touch the probe itself, to catch bit-rot in the bring-up recipes.
Author
CI on
|
| Cell | master eb99fc6 |
this PR d0c387e |
|---|---|---|
beegfs-7.4.6-git-annex |
❌ | ❌ |
beegfs-7.4.6-pjdfstest |
❌ | ❌ |
beegfs-8.1.0-git-annex |
❌ | ❌ |
beegfs-8.1.0-pjdfstest |
❌ | ❌ |
loop-ext4-git-annex |
❌ | ❌ |
loop-vfat-git |
❌ | ❌ |
loop-vfat-git-annex |
❌ | ❌ |
loop-vfat-pjdfstest |
❌ | ❌ |
loop-vfat-stress-ng |
❌ | ❌ |
nfs-pjdfstest |
❌ | ❌ |
beegfs-7.4.6-git, beegfs-7.4.6-stress-ng |
✅ | ✅ |
beegfs-8.1.0-git, beegfs-8.1.0-stress-ng |
✅ | ✅ |
loop-ext4-git, loop-ext4-pjdfstest, loop-ext4-stress-ng |
✅ | ✅ |
nfs-git, nfs-git-annex, nfs-stress-ng |
✅ | ✅ |
An exact match in both directions: no cell that is green on master is red here, and this PR turns nothing red that was not already red. matrix and codespell pass; publish is skipped, as intended on a pull request.
Most of the red is documented in GOTCHAS.md and is the point of the row rather than a defect — all four Loop vfat cells (no symlinks, no ownership, and a filename charset that git's own suite deliberately violates), and the pjdfstest cells. Two are worth chasing on their own, and neither belongs to this PR:
loop-ext4-git-annex— ext4 is the control row, soGOTCHAS.mdalready calls this a real bug rather than a filesystem property. It ran with a 100 MB image here (loop3: detected capacity change from 0 to 204800in dmesg), i.e. the per-filesystem size floor added in this PR is inert for ext4 and did not change what the cell does.- The BeeGFS
git-annexpair, red onmastersince before this branch.
I have not spent a re-run on any of these: comparing against master's own per-cell status is stronger evidence than a second attempt at a cell that is documented as reliably red.
One correction to the PR description above, which was written before this branch was merged with master: _test-under.yaml is not "updated" — master deleted it when the per-cell dispatchers were replaced by the single matrix workflow, so reproduce.yaml now carries those steps directly. Likewise capabilities is defined in .github/matrix.yaml (flagged on-demand: true), not in matrix.sh; the grid stays at exactly 20 cells.
Generated by Claude Code
master's PR #10 replaced every backend's fixed /mnt/<backend> with a per-run directory derived from the same mktemp base as the backend's backing state. This branch added a backend while that was landing, so git merged bin/eval-under-sshfs cleanly and left it on /mnt/sshfs -- textually fine, and wrong: the README this merge brings in states that a backend mounts next to its backing state "never a fixed /mnt/<backend>", and its "Adding a new backend" contract names $MNT_BASE.<backend> outright. The merge is only done once the new backend obeys it. So eval-under-sshfs now derives both paths from one mktemp -u base: /tmp/eval-under-sshfs-B3UXj.scratch host key, sshd conf, backing dir /tmp/eval-under-sshfs-B3UXj.sshfs where this run mounts it and teardown removes the mountpoint only when this run created it, so an explicit --mount-point that predates us is unmounted rather than deleted -- same rule the other backends now follow. The 755 on the scratch dir (mktemp -d would give 700) now says why it is there: sshd reads its config as root while the mount runs as the invoking user. README's conflict was two additions to the same spot, not a disagreement: master's mountpoint explanation follows the CLI flag list it belongs to, then the Reproducing-a-report section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK
An independent shell review found that this backend has never worked in
the mode it documents. Everything I had tested ran as root *directly*,
which takes the branch that needs no privilege drop; under
`sudo bin/eval-under sshfs ...` -- the invocation in its own --help, and
the one bin/ci/run-under.sh uses -- it died instantly:
bin/eval-under-sshfs: line 190: -u: command not found (exit 127)
`as_invoker` and the final dispatch both wrote `"${SUDO[@]}" -u USER`,
and SUDO is *empty* exactly when we are already root and therefore
actually need the drop, so the expansion began with a bare `-u`.
bin/eval-under-nfs spells out `sudo -u` for this reason; the array idiom
was copied into a place where it cannot work. Both call sites now do the
same, and the sudo-availability check applies whenever a drop is needed
rather than only when we need sudo to become root.
Reproduced and fixed with it, all verified against a real unprivileged
invoker (f=3 g=4 nlink=1 once again, this time not as root):
- The scratch dir was created by root and only chmod'd, never chowned,
so every as_invoker write into it (both keys, authorized_keys, the
backing dir) was EACCES the moment the first bug was out of the way.
- UsePAM no made sshd apply its own locked-account check, refusing any
invoker whose shadow password field is "!" -- the normal state of a CI
or service account -- with nothing but "Connection reset by peer" to
explain it. PAM is on now, password auth stays off, and sshd keeps a
log (-E) that is dumped when a mount fails, because an auth refusal is
explained nowhere else.
- The free-port scan promised concurrent runs would not collide and did
not deliver: four at once, three died on "sshd did not start". A lost
bind is now retried on the next free port. Measured: 4/4 succeed on
2223-2226, no leaks.
- A signal tore the filesystem down under a still-running wrapped
command, leaving it writing into a deleted inode. INT/TERM/HUP now
reach teardown, which kills the command before unmounting.
- An explicit --mount-point was chown'd to the invoker and never
restored; only a directory this run created is re-owned now.
- A pinned --port could leak an sshd that started after the pid-file
wait expired, since SSHD_PID was still empty.
- --user without --host is now rejected instead of hanging on an
AllowUsers mismatch.
Outside the backend:
- bin/eval-under-loop: --size 40M slipped past the floor comparison (a
failing test inside `if` is exempt from set -e) into dd count=40M,
which GNU dd reads as 40 TiB; the floor check was the only thing that
looked like validation, so now it is.
- bin/ci/fs-capabilities.sh: without python3, seven checks reported as
missing *filesystem features* -- including sqlite-wal, which the header
calls the most predictive value here -- so a profile pasted into a bug
report blamed the filesystem for a missing interpreter. python3 is now
a preflight, and an absent helper reports unknown, never no.
- bin/ci/probe-backend.sh: the cifs probe appended a mode-777
"force user = root" share to smb.conf and left a known SMB password for
root in the passdb, reverting neither, so the next `systemctl start
smbd` re-exposed it on every interface; the share moves to an included
file that teardown removes, is bound to loopback, and the password is
taken back out. The sshfs probe appended to the real
~/.ssh/authorized_keys and never removed it -- precisely what
eval-under-sshfs exists to avoid -- so it now removes its own line and
key. defer() quoted nothing, so a probe dir containing a space leaked
its mount while still reporting success; cleanups now carry quoted
argv, with defer_shell for the few that need shell syntax. And an
unusable EVAL_UNDER_PROBE_DIR was recorded as "NOT BOOTSTRAPPABLE"
with exit 0, contradicting this script's own closing comment: a broken
runner is not a filesystem finding, and now exits 2.
- reproduce.yaml: dispatch inputs reach the shell through env instead of
being template-substituted into run: bodies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK
An independent fact-check of the documentation found the central claim about git-annex on sshfs to be wrong, in the direction that understates the problem. GOTCHAS.md said locked repos and v10 unlocked (pointer-file) repos were both fine and that only the adjusted unlocked branch broke. Re-measured with the same git-annex 10.20240129: A plain v10, locked add OK B plain v10, annex.addunlocked=true FAILED -> failed to link to annex C plain v10, git add via annex.largefiles OK So the boundary is not locked-vs-unlocked and not adjusted-vs-plain: it is who performs the ingest. git-annex hardlinking content into the annex fails; git's own filter writing a pointer does not. The earlier wording came from reading git-annex's suite, where `Repo Tests v10 unlocked` is green -- but that group is green because the suite's unlocked mode adds with `git add`. DataLad calls `git annex add`, so plain v10 unlocked repos break for it too, not just adjusted ones. The file now says which inference was wrong, because it is an easy one to repeat. Code, where the docs turned out to describe behaviour the code did not have: - update-status.py / render-report.py read matrix.yaml's targets unfiltered, so the publish job would have added 5 permanently-"unknown" *-capabilities badges and a 5th never-fillable column to the report page -- while FILESYSTEMS.md and matrix.yaml both promise an on-demand target costs "no badge". Both now apply the same on-demand filter as matrix-json.sh. Measured after the fix: 20 cells, 21 badges (was 25/26), no capabilities column. - run-under.sh now refuses sshfs + a needs-root target. reproduce.yaml offers sshfs x pjdfstest and sshfs x stress-ng in dropdowns, but the sshfs backend always drops to the invoking user and has no --no-root-squash counterpart, so those cells could only ever measure the absence of privilege. Its unknown-target message also listed only the scheduled targets while accepting `capabilities`. - The probe roll-up's 13 lines of inline shell and awk violated .claude/CLAUDE.md rule 1; extracted to bin/ci/render-probe-table.sh, which also now explains an empty summaries dir instead of rendering a headerless table. Job count in the header corrected (24 probe + 3 smoke + 1 roll-up = 28, not "~25"). Documentation corrections, each re-derived rather than taken on trust: - The sshfs mechanism is sharper: the link IS real server-side (one inode, nlink=2, in the backing dir); what SFTP cannot carry is identity, its attribute record having neither an inode number nor a link count. `use_ino` would not have helped under FUSE2 either -- it only passed through inode numbers a filesystem supplies, and sshfs has none. Added the measured stale-size effect: right after writing one name, the other reads back as size 0, with -o cache=no too. - "Off is the honest mount" overstated -o cache=no: it disables sshfs's own cache, not the kernel's 1-second attribute timeout, which is why that stale size survives it. - The gfs2/ocfs2 size floor's stated reason was measurably false: with this backend's -J 8 both filesystems DO fit in 100MB. The floor is about leaving `git annex test` room, which is what it now says. - Vendor policy was misattributed: Red Hat does not document single-node GFS2 as test-only, it does not support it at all outside backup/DR; Oracle documents local OCFS2 as a supported, migratable configuration. - The Lustre ubuntu2404 claim was stale. `latest-release` (2.15.x LTS, what the probe queries) publishes no ubuntu2404 client repo, but the 2.16.x feature line does. The conclusion survives -- those modules are built for 6.8.0-35-generic, not the runner's azure kernel -- so the reason is narrower than "no repo exists". - The ntfs3 recommendation was not implementable as written: there is no mkfs.ntfs3, install-backend.sh has no ntfs entry, and eval-under-loop's ntfs branch would land on ntfs-3g's FUSE driver rather than the kernel one. Says so now, and notes the profile came from a Linux-made volume, not a Windows-formatted disk. - Part 1 says SQLite WAL fails on Gluster while Part 2 measures sqlite-wal=yes; reconciled rather than left to look like a refutation. The probe's single-brick localhost volume is not the multi-brick deployment the 2015 report came from. - Scoped the "~/.ssh/authorized_keys is never written to" guarantee to the backend: probe-backend.sh does use the system sshd and does append to it (removing its line on teardown as of the previous commit). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK
Author
Updated after review: CI on
|
| Cell | master aa98679 |
this PR b917d58 |
|---|---|---|
beegfs-7.4.6-git-annex, beegfs-7.4.6-pjdfstest |
❌ | ❌ |
beegfs-8.1.0-git-annex, beegfs-8.1.0-pjdfstest |
❌ | ❌ |
loop-ext4-git-annex |
❌ | ❌ |
loop-vfat-git, loop-vfat-git-annex, loop-vfat-pjdfstest, loop-vfat-stress-ng |
❌ | ❌ |
nfs-pjdfstest |
❌ | ❌ |
beegfs-7.4.6-git, beegfs-7.4.6-stress-ng |
✅ | ✅ |
beegfs-8.1.0-git, beegfs-8.1.0-stress-ng |
✅ | ✅ |
loop-ext4-git, loop-ext4-pjdfstest, loop-ext4-stress-ng |
✅ | ✅ |
nfs-git, nfs-git-annex, nfs-stress-ng |
✅ | ✅ |
20/20 identical in both directions: nothing green on master is red here, and nothing red here was green there. matrix and codespell pass; publish is skipped, as intended on a pull request. No re-run spent — comparing against master's own per-cell status is better evidence than a second attempt at a cell documented as reliably red.
The reds are the point of those rows, not defects in them: all four Loop vfat cells (no symlinks, no ownership, and a filename charset git's own suite deliberately violates) plus the pjdfstest cells, all documented in GOTCHAS.md. Two are worth chasing separately and neither belongs to this PR — loop-ext4-git-annex (ext4 is the control row, so GOTCHAS.md already calls it a real bug) and the BeeGFS git-annex pair, red on master since before this branch.
What the reviews changed
Two things are worth flagging to a reviewer of this PR specifically, because they were wrong in ways CI could not have caught:
- The sshfs backend had never worked in the mode it documents. Under
sudo bin/eval-under sshfs …— its own--helpexample, and whatbin/ci/run-under.shuses — it exited 127 on"${SUDO[@]}" -u, because that array is empty exactly when we are already root and need the privilege drop. Every earlier test had run as root directly, which takes the other branch. Fixed and re-verified against a real unprivileged invoker. - A false claim at the centre of the documentation.
GOTCHAS.mdsaid v10 unlocked repos were fine and only the adjusted unlocked branch broke. Re-measured: a plain v10 repo withannex.addunlocked=truefails identically. The boundary is who ingests — git-annex hardlinking into the annex fails, git's filter writing a pointer does not. The greenRepo Tests v10 unlockedgroup in git-annex's suite is green because that mode adds withgit add; generalising from it was the error. This matters for DataLad, which callsgit annex add.
Also fixed: the publish side would have written 5 permanently-unknown *-capabilities badges and a never-fillable column, contradicting the "no badge" promise for on-demand targets (now 20 cells / 21 badges, measured); reproduce.yaml offered sshfs × root-requiring targets that cannot work; a --size 40M reached dd count=40M (40 TiB); fs-capabilities.sh reported a missing python3 as missing filesystem features, sqlite-wal included; and the cifs probe left a mode-777 force user = root Samba share plus a root SMB password behind, reverting neither.
Full detail is in the two commit messages (e46b3ce, b917d58).
Generated by Claude Code
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive filesystem capability measurement and introduces sshfs as a new eval-under backend, enabling testing against FUSE-based and remote filesystems.
Key Changes
New Filesystem Probing Infrastructure
bin/ci/probe-backend.sh: Reconnaissance tool that attempts to stand up candidate filesystems on GitHub-hosted runners and measures their POSIX capabilities. Supports 21 filesystem candidates (gfs2, ocfs2, f2fs, exfat, ntfs3, udf, nilfs2, bcachefs, zfs, overlay, glusterfs, cephfs, cifs, sshfs, gocryptfs, encfs, ecryptfs, s3-rclone, lustre-client, openafs, vm-only).bin/ci/fs-capabilities.sh: Fast capability profiler (~1 second) that checks for POSIX features known to break git-annex/DataLad: SQLite WAL support, fcntl locks, link() EEXIST semantics, symlinks, hardlink inode consistency, exec bits, permission bits, rename-over-existing, unlink-open behavior, and flock support. Output is machine-readable key=value pairs..github/workflows/probe-filesystems.yaml: CI workflow that runs probes across candidate filesystems on ubuntu-24.04 (and ubuntu-22.04 for kernel-module-dependent candidates). Collects capability profiles and generates a roll-up summary.New sshfs Backend
bin/eval-under-sshfs: Full-featured sshfs backend with two modes:--host): Mounts a directory on a named host using existing ssh config/agent, for reproducing reporter-specific setups.--no-cacheto disable attribute caching,--workaroundfor sshfs compatibility switches (e.g.,renamefor non-atomic rename-over-existing), custom mount options via--opt.Documentation
FILESYSTEMS.md: Comprehensive survey mapping real user-reported git-annex/DataLad failures to their root causes (SQLite locking, POSIX locks, link() semantics, crippled filesystems) and cross-referencing with probe results. Documents which filesystems have been measured and what they support.GOTCHAS.md(updated): Clarifies mkfs/mount settings for cluster filesystems (gfs2, ocfs2) using single-node locking modes, image size floors for journal overhead, and the approximation nature of testing cluster filesystems without a cluster.Framework Extensions
bin/eval-under-loop(updated): Added support for gfs2, ocfs2, f2fs, exfat, nilfs2 with appropriate mkfs options and image size floors. New--mkfs-optsflag to override defaults.bin/ci/install-backend.sh(updated): Added sshfs installation logic.bin/ci/run-under.sh(updated): Integrated sshfs backend into the runner orchestration.bin/ci/target-capabilities.sh: New on-demand target that runs the capability probe inside an eval-under wrapper..github/workflows/_test-under.yaml(updated): Addedbackend-optionsandrunnerinputs to support on-demand reproduction with custom mount options..github/workflows/reproduce.yaml: New on-demand workflow for reproducing specific filesystem+target combinations without waiting for scheduled runs.bin/ci/matrix.sh(updated): DefinedEVAL_UNDER_ONDEMAND_TARGETSto distinguish capabilities (fast, on-demand) from scheduled matrix targets.README.md(updated): Added references to FILESYSTEMS.md and examples of sshfs usage.https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK