From a2ab6d68b5514f73707724e3def0ba363f48f192 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 20:49:48 +0000 Subject: [PATCH 1/4] Add an sshfs backend to eval-under sshfs is the filesystem people actually reach for when they mount a remote over ssh, and it breaks git-annex in a way no local filesystem does. This adds it as a backend so a report against it can be reproduced directly. Two modes: - Loopback (default): a throwaway sshd on the first free port at or above 2222, with its own host key, authorized_keys and pid file inside the run's scratch directory, and a fresh backing directory sshfs-mounted back over it. The system sshd is not used and ~/.ssh/authorized_keys is never written to. - `--host` mounts a real remote using the caller's ssh config, so a reporter's own server and mount options can be used verbatim. Knobs for the options reports actually turn on: `--no-cache`, `--workaround`, `--opt`, `--port`, `--user`, `--remote-dir`, plus the common `--mount-point` / `--set-home` / `--keep`. Notes on the two non-obvious pieces: - Dropping privileges deliberately calls `sudo -u` literally rather than going through the `${SUDO[@]}` array the other backends use: that array is empty when we are already root, which is exactly the case that needs the drop. - sshd runs with `UsePAM yes`. With it off, sshd refuses any account whose shadow entry is locked (`!`), which is the normal state for service and CI accounts, and the mount fails for a reason that looks nothing like its cause. Its log is dumped on mount failure for the same reason. `bin/ci/run-under.sh` refuses sshfs together with a root-requiring target (pjdfstest): a FUSE mount belongs to whoever mounted it, there is no --no-root-squash equivalent, and the run would measure privilege rather than the filesystem. No matrix cell and no badge -- the backend exists for on-demand reproduction, and GOTCHAS.md records what it does to hardlink identity, mtime granularity and fifos before anyone reads a red result as a filesystem bug. Split out of the filesystem-probing branch, which had grown too large to review as one change. Verified: shellcheck clean, all 28 bats tests pass (the four "every installed backend" cases now cover this backend), the mount works as root and via the privilege-drop path, and the refusal above exits 2. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK --- GOTCHAS.md | 111 +++++++++ README.md | 9 + bin/ci/install-backend.sh | 20 +- bin/ci/run-under.sh | 18 +- bin/eval-under-sshfs | 506 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 660 insertions(+), 4 deletions(-) create mode 100755 bin/eval-under-sshfs diff --git a/GOTCHAS.md b/GOTCHAS.md index c2e7675..8c02138 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -75,6 +75,117 @@ So `--no-root-squash` (env: `EVAL_UNDER_NFS_NO_ROOT_SQUASH`) exports with user would. The loop and BeeGFS backends already run the wrapped command as root, so the flag is a no-op there. +### sshfs (`bin/eval-under-sshfs`) + +A throwaway sshd is started on the first free port at or above 2222 -- +its own host key, its own `authorized_keys`, its own pid file, all inside +the run's scratch directory -- and a fresh backing directory is +sshfs-mounted back over it. The system sshd is not used and +`~/.ssh/authorized_keys` is never written to. With `--host` it mounts a +real remote instead, using the caller's ssh config. + +| Knob | Value | Why | +| --- | --- | --- | +| Port | first free `>= 2222` | Two runs at once (a reproduction while a suite is going, two CI cells on one runner) must not collide. `--port` pins it. | +| `-o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3` | always | A dropped connection should fail the command, not wedge it forever. | +| `-o cache=no` | only with `--no-cache` | sshfs caches attributes by default, which hides stale-stat behaviour. This turns off sshfs's own cache, not all caching -- the kernel's 1-second attribute timeout still applies, and the stale-size effect above survives it. On is what users actually have. | +| `-o workaround=rename` | only with `--workaround rename` | Makes sshfs emulate rename-over-existing by unlinking first -- non-atomic, which is what SFTP servers without the POSIX-rename extension force. | +| Mount/command user | the invoking user | A FUSE mount belongs to whoever ran `sshfs`; root cannot read it without `allow_other`. Same treatment `eval-under-nfs` gives `root_squash`. | + +**Hardlinks exist but are not observable, and that is the whole story +for git-annex.** `ln a b` succeeds over SFTP, and then `a` and `b` report +*different* inode numbers and `nlink=1` each: + +``` +ext4 name=a ino=1884275 nlink=2 name=b ino=1884275 nlink=2 +sshfs name=a ino=3 nlink=1 name=b ino=4 nlink=1 +``` + +This is not a misconfiguration, and the link is not fake: in the backing +directory on the server both names really do share one inode with +`nlink=2`. What SFTP cannot carry is *identity* -- its attribute record +has neither an inode number nor a link count -- so sshfs synthesises an +`st_ino` per path and reports `nlink=1` for everything. There is no knob +for it: `use_ino` (removed in libfuse 3) only ever passed through inode +numbers that a filesystem supplies, and sshfs has none to supply, so it +would not have helped under FUSE2 either; sshfs 3.7 offers only +`disable_hardlink`, which makes `link()` fail outright. + +Worse than invisible, and worth knowing when a report mentions truncated +files: immediately after writing one name, the *other* name still reads +back with size 0 through the mount -- with `-o cache=no` as well. The +capability probe reports the inode half as +`hardlink-same-inode=no` / `hardlink-nlink=no` while `hardlink=yes` -- +which is exactly why those two checks exist. `hardlink` alone called +sshfs healthy. + +What it costs, measured with git-annex 10.20240129: + +- `git annex add` on a **locked** branch: fine. The file becomes a + symlink into `.git/annex/objects`, and symlinks work. +- **Any unlocked `git annex add`: fails** -- `foo failed to link to + annex`. add hardlinks the content into the annex and then verifies the + link, and the verification cannot succeed on a filesystem where the + link is invisible. This is not limited to an adjusted branch: a plain + v10 repo fails identically with `annex.addunlocked=true` in git + config, set through `git annex config`, or passed as `-c`. +- `git add` through git's own filter (with `annex.largefiles` matching): + **fine** -- the pointer is committed and `git annex fsck` is clean. + So is `git annex unlock` of an already-committed file. +- `git clone` of a local repo: **fails** -- + `fatal: hardlink different from source at '...'`. git's local-clone + path hardlinks objects and runs the same check. + +So the boundary is not locked-vs-unlocked, and not adjusted-vs-plain: +it is **who does the ingest**. git-annex hardlinking content into the +annex fails; git's filter writing a pointer does not. + +That distinction is easy to get backwards from the test suite alone, +because `git annex test` shows `Repo Tests v10 unlocked` **green** and +only `v10 adjusted unlocked branch` red (11 of 12 in its Init Tests +group, once `add` fails). The green group is not evidence that unlocked +repos are fine here -- the suite's unlocked mode ingests with `git add`. +An earlier revision of this file drew exactly that inference and was +wrong. + +It matters for DataLad, which calls `git annex add`: plain v10 unlocked +repos break for it too, not just adjusted ones. + +**`git annex test` wedges partway through, reproducibly.** Both +full-suite runs stopped at the same place -- `Remote Tests / unavailable +remote / removeKey` -- and sat there until killed (15+ minutes on the +second). On ext4 that same test takes **0.02s and passes**, and the whole +suite finishes in 1m21s. + +It is not an I/O hang. While wedged: + +- the mount stays responsive (`ls` returns immediately), +- git-annex holds **no open files on the mount and no sockets**, +- its threads sit in `futex_do_wait` / `ep_poll`, with no child + processes outstanding. + +That is a process waiting on something internal, not one blocked on the +filesystem. Note also that git-annex sets `annex.sshcaching = false` here +on its own, because ssh control sockets need unix sockets and this mount +has none -- so the run is already on a different code path from a normal +one. + +Two caveats before anyone reports this upstream: the same test passes in +seconds when selected on its own with `-p '/unavailable remote/'`, so it +needs the full-suite context; and this was git-annex 10.20240129 from +Ubuntu 24.04, not a daily build. Re-run it through the reproduce +workflow, which installs the daily build from con/git-annex, before +filing anything. + +**Timestamps are quantised to the second.** Five files created back to +back get one or two distinct mtimes, where every other filesystem +measured gives five. Nothing else in the matrix has a clock this coarse, +which makes sshfs the row that exercises git's racy-timestamp handling. + +**No fifos, no unix sockets.** git-annex says so itself at init +("Detected a filesystem without fifo support") and adapts. Worth knowing +before reading it as a failure. + ### BeeGFS (`bin/eval-under-beegfs`) A containerised cluster (`fixtures/beegfs/docker-compose-v{7,8}.yml`) plus diff --git a/README.md b/README.md index 998c765..3221e0b 100644 --- a/README.md +++ b/README.md @@ -145,6 +145,14 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \ # the fsync-heavy slow path) sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test' +# Under sshfs, with the attribute cache off +sudo bin/eval-under sshfs --no-cache --set-home -- \ + bash -c 'cd "$HOME" && git annex test' + +# ...or against the reporter's own server, with their mount options +sudo bin/eval-under sshfs --host store.example.org --remote-dir /data/scratch \ + --workaround rename --set-home -- git annex fsck + # Skip teardown to poke around after a failure sudo bin/eval-under beegfs --set-home --keep -- some-failing-command @@ -188,6 +196,7 @@ into `bin/eval-under`, bumped with each release tag. | `bin/eval-under` | Dispatcher: routes to `bin/eval-under-` | | `bin/eval-under-beegfs` | BeeGFS backend (containerised cluster + kernel client mount) | | `bin/eval-under-nfs` | NFS backend (localhost loopback export) | +| `bin/eval-under-sshfs` | sshfs backend (throwaway loopback sshd, or a remote you name) | | `bin/eval-under-loop` | Loop-device backend (dd + losetup + mkfs. + mount) | | `fixtures/beegfs/docker-compose-v7.yml` | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host` | | `fixtures/beegfs/docker-compose-v8.yml` | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane) | diff --git a/bin/ci/install-backend.sh b/bin/ci/install-backend.sh index 6c5ee54..4d76d7a 100755 --- a/bin/ci/install-backend.sh +++ b/bin/ci/install-backend.sh @@ -9,17 +9,18 @@ # usage: # bin/ci/install-backend.sh # -# backend = beegfs | nfs | loop +# backend = beegfs | nfs | loop | sshfs # version = for beegfs: point release (e.g. 7.4.6, 8.1.0) # for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) # for nfs: literal "n/a" +# for sshfs: literal "n/a" # # Idempotent enough for CI re-runs; not a full package manager. set -euo pipefail export DEBIAN_FRONTEND=noninteractive -BACKEND="${1:?backend required (beegfs|nfs|loop)}" +BACKEND="${1:?backend required (beegfs|nfs|loop|sshfs)}" VERSION="${2:?version required (BeeGFS version | loop fs name | 'n/a' for nfs)}" # Give unattended-upgrades a moment on ubuntu-22.04 runners rather than @@ -72,6 +73,18 @@ install_nfs() { command -v exportfs } +install_sshfs() { + apt_update + # openssh-sftp-server is what actually serves the mount; on Ubuntu it + # is pulled in by openssh-server, but name it so a slimmer image + # cannot leave us without an sftp-server binary. + apt_install sshfs openssh-server openssh-sftp-server + command -v sshfs + # The backend starts its own sshd, so the system one need not run -- + # but its privilege-separation directory must exist. + sudo mkdir -p /run/sshd +} + install_loop() { local pkg case "$VERSION" in @@ -90,5 +103,6 @@ case "$BACKEND" in beegfs) install_beegfs ;; nfs) install_nfs ;; loop) install_loop ;; - *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop)" >&2; exit 1 ;; + sshfs) install_sshfs ;; + *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop|sshfs)" >&2; exit 1 ;; esac diff --git a/bin/ci/run-under.sh b/bin/ci/run-under.sh index 8580891..4c69ecf 100755 --- a/bin/ci/run-under.sh +++ b/bin/ci/run-under.sh @@ -13,10 +13,11 @@ # usage: # bin/ci/run-under.sh [target] # -# backend = beegfs | nfs | loop +# backend = beegfs | nfs | loop | sshfs # version = for beegfs: point release (e.g. 7.4.6, 8.1.0) # for loop: filesystem type (e.g. vfat, ext4) # for nfs: literal "n/a" +# for sshfs: literal "n/a" # target = git-annex (default) | git | stress-ng | pjdfstest # # env overrides: @@ -43,6 +44,20 @@ target_known "$TARGET" || { exit 1 } +# A root-requiring suite under a backend that can only run as the +# invoking user would measure nothing: every privileged syscall it exists +# to test fails for want of privilege, not because of the filesystem. The +# NFS backend has --no-root-squash for this; sshfs has no equivalent, +# because a FUSE mount belongs to whoever mounted it. Refuse the pair up +# front rather than produce a meaningless red result. +if [ "$BACKEND" = sshfs ] && target_needs_root "$TARGET"; then + echo "$TARGET needs root, and the sshfs backend always runs the wrapped" >&2 + echo "command as the invoking user (a FUSE mount belongs to its mounter)." >&2 + echo "There is no --no-root-squash equivalent here, so this combination" >&2 + echo "cannot measure what the target is for. Try the nfs or loop backend." >&2 + exit 2 +fi + # The target scripts re-derive their own defaults from matrix.sh, but an # override handed to us must survive into the wrapped child. export EVAL_UNDER_SRC_DIR @@ -59,6 +74,7 @@ case "$BACKEND" in # need an export that does not squash root, and need to keep # their privileges rather than being dropped to the invoker. target_needs_root "$TARGET" && opts=(--no-root-squash) ;; + sshfs) opts=() ;; *) echo "unknown backend: $BACKEND" >&2; exit 1 ;; esac diff --git a/bin/eval-under-sshfs b/bin/eval-under-sshfs new file mode 100755 index 0000000..3c952fa --- /dev/null +++ b/bin/eval-under-sshfs @@ -0,0 +1,506 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-sshfs: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR (and +# optionally HOME) pointing at an sshfs (FUSE-over-SFTP) mount. +# +# Backend of the eval-under framework. +# +# Two modes: +# +# loopback (default) -- brings up a throwaway sshd on a high port, with +# its own host key and its own authorized_keys, and sshfs-mounts a +# fresh backing directory back over it. Nothing outside the scratch +# dir is touched: the user's ~/.ssh is never written to. +# +# remote (--host) -- sshfs-mounts a directory on a host you name, using +# your existing ssh config/agent. For reproducing a reporter's +# actual setup, where the interesting variable is often their +# server's SFTP implementation rather than sshfs itself. + +set -eu + +# Empty MNT means "derive a per-run mountpoint next to the scratch dir" +# (see the MNT_BASE block below); only an explicit --mount-point / +# EVAL_UNDER_MOUNT pins the mount to a fixed path. +MNT="${EVAL_UNDER_MOUNT:-}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +# Loopback-mode knobs. An empty PORT means "pick a free one": two +# eval-under runs on the same machine (or two CI cells on one runner) +# must not fight over a fixed port. +PORT="${EVAL_UNDER_SSHFS_PORT:-}" +# Remote mode: unset means loopback. +HOST="${EVAL_UNDER_SSHFS_HOST:-}" +REMOTE_USER="${EVAL_UNDER_SSHFS_USER:-}" +REMOTE_DIR="${EVAL_UNDER_SSHFS_DIR:-}" + +# sshfs -o options. EVAL_UNDER_SSHFS_OPTS is a comma-separated string +# appended after ours, so a caller can override any default we set. +EXTRA_OPTS="${EVAL_UNDER_SSHFS_OPTS:-}" +NO_CACHE="${EVAL_UNDER_SSHFS_NO_CACHE:-0}" +WORKAROUND="${EVAL_UNDER_SSHFS_WORKAROUND:-}" + +usage() { + cat <<'EOF' +Usage: eval-under-sshfs [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at an sshfs mount. + +By default this is a loopback mount: a throwaway sshd is started on a +high port with its own host key and its own authorized_keys file, and a +fresh backing directory is sshfs-mounted back over it. Your ~/.ssh is +not read or written. With --host, mounts a remote directory instead, +using your normal ssh configuration. + +Requires the `sshfs` package (and, for loopback mode, `openssh-server`). +Install with: apt install sshfs openssh-server + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT (next to scratch dir) + Where to mount sshfs on the host. Default: a fresh per-run + directory next to this run's scratch dir, both under $TMPDIR + (/tmp if unset) -- e.g. with TMPDIR=~/.tmp: + + ~/.tmp/eval-under-sshfs-B3UXj.scratch keys, sshd conf, backing + ~/.tmp/eval-under-sshfs-B3UXj.sshfs where this run mounts it + + A fixed default would collide between concurrent runs and hide + which run owned the mount. Pass this option when a stable path is + wanted. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Skip teardown; leave the mount (and the throwaway sshd) up for + debugging. + + --port N EVAL_UNDER_SSHFS_PORT (first free >=2222) + Port for the throwaway sshd (loopback mode only). Left unset, the + first free port at or above 2222 is used, and a port lost to a + concurrent run between the scan and the bind is retried on the + next one up, so concurrent runs do not collide. + + --host HOST EVAL_UNDER_SSHFS_HOST (unset) + Mount from a remote host instead of starting a local sshd. Uses + your ssh config, keys and agent as-is. Implies that --port refers + to that host's sshd. + + --user USER EVAL_UNDER_SSHFS_USER (invoking user) + Remote username. Requires --host: the throwaway sshd generated for + loopback mode only admits the invoking user, so any other name + there could only ever be refused. + + --remote-dir PATH EVAL_UNDER_SSHFS_DIR (remote: required) + Directory on the remote host to mount. In loopback mode this + defaults to a fresh scratch directory and should be left alone. + + --no-cache EVAL_UNDER_SSHFS_NO_CACHE (unset) + Pass `-o cache=no`. sshfs caches attributes and directory listings + by default, which papers over exactly the stale-stat behaviour a + reporter may be hitting. Turn the cache off to see the raw + protocol semantics. + + --workaround LIST EVAL_UNDER_SSHFS_WORKAROUND (unset) + Pass `-o workaround=LIST` (sshfs's own compatibility switches, + e.g. `rename`, `truncate`, `buflimit`). `rename` is the + interesting one here: it makes sshfs emulate rename-over-existing + by unlinking the destination first, which is what it has to do + against SFTP servers lacking the POSIX-rename extension -- a + non-atomic rename, and precisely the semantics git and git-annex + assume they can rely on. + + --opt OPTS EVAL_UNDER_SSHFS_OPTS (unset) + Extra comma-separated `-o` options, appended last so they win. + May be given more than once. + + -h, --help + Print this help. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + +Exit status: the wrapped command's exit status. Teardown runs on any +exit (unless --keep). + +Examples: + + # git-annex test over loopback sshfs, cache off + sudo bin/eval-under sshfs --no-cache --set-home -- \ + bash -c 'cd "$HOME" && git init t && cd t && git annex init && git annex test' + + # Reproduce non-atomic rename-over-existing + sudo bin/eval-under sshfs --workaround rename --set-home -- some-command + + # Against a reporter's own server + sudo bin/eval-under sshfs --host store.example.org --remote-dir /data/scratch \ + --set-home -- git annex fsck +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --port) PORT="$2"; shift 2 ;; + --host) HOST="$2"; shift 2 ;; + --user) REMOTE_USER="$2"; shift 2 ;; + --remote-dir) REMOTE_DIR="$2"; shift 2 ;; + --no-cache) NO_CACHE=1; shift ;; + --workaround) WORKAROUND="$2"; shift 2 ;; + --opt) EXTRA_OPTS="${EXTRA_OPTS:+$EXTRA_OPTS,}$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ $# -gt 0 ] || { echo "no command given" >&2; usage >&2; exit 2; } + + +# Did the caller pin the port? Decided before start_local_sshd starts +# reassigning PORT while hunting for a free one. +PORT_PINNED="${PORT:+1}" + +if [ "$(id -u)" -ne 0 ]; then + command -v sudo >/dev/null || { + echo "must run as root (or have sudo available)" >&2; exit 2; } + SUDO=(sudo) +else + SUDO=() +fi + +# The FUSE mount belongs to whoever runs sshfs, and the wrapped command +# has to be able to read it. Both therefore run as the invoking user -- +# the same treatment bin/eval-under-nfs gives root_squash. +if [ -n "${SUDO_UID:-}" ]; then + INVOKER_UID="$SUDO_UID"; INVOKER_GID="${SUDO_GID:-$SUDO_UID}" + INVOKER_USER="${SUDO_USER:-$INVOKER_UID}" +else + INVOKER_UID="$(id -u)"; INVOKER_GID="$(id -g)" + INVOKER_USER="$(id -un)" +fi + +# In loopback mode the generated sshd config has AllowUsers , so +# a different --user could only ever be refused -- with nothing but +# "Connection reset by peer" to explain it. +if [ -n "$REMOTE_USER" ] && [ -z "$HOST" ]; then + echo "--user needs --host: the throwaway sshd only accepts $INVOKER_USER" >&2 + exit 2 +fi + +# Dropping privileges needs `sudo` itself, so this deliberately does NOT +# go through ${SUDO[@]}: that array is EMPTY when we are already root, +# which is exactly the case that needs the drop. Under +# `sudo eval-under sshfs ...` -- the documented invocation, and the one +# bin/ci/run-under.sh uses -- id -u is 0 while INVOKER_UID is the real +# user, so "${SUDO[@]}" -u would expand to a bare `-u`. bin/eval-under-nfs +# spells out `sudo -u` for the same reason. +as_invoker() { + if [ "$INVOKER_UID" = "$(id -u)" ]; then + "$@" + else + sudo -u "$INVOKER_USER" "$@" + fi +} + +# Dropping privileges is only possible with sudo present, whether or not +# we needed it to become root in the first place. +if [ "$INVOKER_UID" != "$(id -u)" ]; then + command -v sudo >/dev/null || { + echo "ERROR: need sudo to run as the invoking user ($INVOKER_USER)" >&2 + exit 2 + } +fi + +log() { printf '\nI: %s\n' "$*"; } + +command -v sshfs >/dev/null 2>&1 || { + echo "ERROR: sshfs not installed. Install with: apt install sshfs" >&2 + exit 3 +} + +# The scratch dir (throwaway host key, sshd config, authorized_keys and +# the backing directory that gets re-exported) and the mountpoint derive +# from one mktemp -u base, so a run leaves a self-describing pair of +# siblings under $TMPDIR -- the convention every backend here follows +# (see "Adding a new backend" in the README). +MNT_BASE="$(mktemp -u "${TMPDIR:-/tmp}/eval-under-sshfs-XXXXX")" +SCRATCH="$MNT_BASE.scratch" +MNT="${MNT:-$MNT_BASE.sshfs}" +mkdir -p "$SCRATCH" +# 755, not mktemp's 700: sshd reads its config and authorized_keys from +# here as root while the mount and the wrapped command run as the +# invoking user. The mode is not enough on its own -- root created the +# directory, so the invoker also has to own it or every as_invoker write +# into it (the keys, authorized_keys, the backing dir) is EACCES. +chmod 755 "$SCRATCH" +chown "$INVOKER_UID:$INVOKER_GID" "$SCRATCH" +SSHD_PID="" + +# Whether teardown created the mountpoint and may therefore remove it. +MNT_CREATED=0 + +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving mount up (exit=$rc)" + echo "I: mount: $MNT" + echo "I: scratch: $SCRATCH" + [ -n "$SSHD_PID" ] && echo "I: sshd pid: $SSHD_PID (kill it yourself)" + return "$rc" + fi + log "teardown" + # Kill the wrapped command first. On a signal it is still running, and + # it must not survive the filesystem it is working on. + pkill -P $$ 2>/dev/null + if command -v fuser >/dev/null 2>&1; then + "${SUDO[@]}" fuser -k -M "$MNT" 2>/dev/null + fi + # fusermount as the mount's owner; fall back to a lazy root umount. + as_invoker fusermount3 -u "$MNT" 2>/dev/null \ + || as_invoker fusermount -u "$MNT" 2>/dev/null \ + || "${SUDO[@]}" umount -l "$MNT" 2>/dev/null + # Kill the whole process group: the daemon forks a child per + # connection, and killing only the listener leaves those behind + # holding the port. + [ -n "$SSHD_PID" ] && { "${SUDO[@]}" kill -- "-$SSHD_PID" 2>/dev/null \ + || "${SUDO[@]}" kill "$SSHD_PID" 2>/dev/null; } + # Only remove the mountpoint if this run made it: an explicit + # --mount-point may well be a directory that predates us. + if [ "$MNT_CREATED" = 1 ]; then + "${SUDO[@]}" rm -rf "$MNT" 2>/dev/null + fi + "${SUDO[@]}" rm -rf "$SCRATCH" 2>/dev/null + return "$rc" +} +# Teardown on a signal too, not just a clean exit: otherwise a SIGTERM +# unmounts and rm -rf's the filesystem out from under a wrapped command +# that is still running, and the orphan keeps writing into a deleted +# inode. on_signal drops the EXIT trap so teardown runs exactly once. +on_signal() { + trap - EXIT + teardown + exit 130 +} +trap teardown EXIT +trap on_signal INT TERM HUP + +# First free TCP port at or above $1 on loopback. `ss` if we have it, +# otherwise a bash /dev/tcp connect probe: a refused connection means +# nothing is listening. +find_free_port() { + local p="$1" limit=$(( $1 + 200 )) + while [ "$p" -lt "$limit" ]; do + if command -v ss >/dev/null 2>&1; then + ss -ltnH "sport = :$p" 2>/dev/null | grep -q . || { echo "$p"; return 0; } + else + if ! (exec 3<>"/dev/tcp/127.0.0.1/$p") 2>/dev/null; then + echo "$p"; return 0 + fi + fi + p=$(( p + 1 )) + done + return 1 +} + +find_sftp_server() { + local p + for p in /usr/lib/openssh/sftp-server \ + /usr/libexec/openssh/sftp-server \ + /usr/lib/ssh/sftp-server \ + /usr/libexec/sftp-server; do + [ -x "$p" ] && { echo "$p"; return 0; } + done + return 1 +} + +# Bring up an sshd that exists only for this run: own port, own host key, +# own authorized_keys, own pid file. Deliberately NOT the system sshd -- +# we would otherwise have to append to the user's authorized_keys and +# hope teardown removes it again. +start_local_sshd() { + local sftp_server sshd_bin + sshd_bin="$(command -v sshd || echo /usr/sbin/sshd)" + [ -x "$sshd_bin" ] || { + echo "ERROR: sshd not found. Install with: apt install openssh-server" >&2 + exit 3 + } + sftp_server="$(find_sftp_server)" || { + echo "ERROR: no sftp-server binary found (openssh-sftp-server?)" >&2 + exit 3 + } + + as_invoker ssh-keygen -t ed25519 -N '' -q -f "$SCRATCH/id" -C eval-under-sshfs + as_invoker ssh-keygen -t ed25519 -N '' -q -f "$SCRATCH/hostkey" -C eval-under-sshfs-host + as_invoker cp "$SCRATCH/id.pub" "$SCRATCH/authorized_keys" + chmod 600 "$SCRATCH/authorized_keys" + + # sshd refuses to start without its privilege-separation directory, + # which is absent in minimal containers (it is created at boot by the + # packaging, not by the package install). + "${SUDO[@]}" mkdir -p /run/sshd + + # Finding a free port and binding it are two steps, and a concurrent + # run can take the port in between -- with four runs on one machine, + # three used to die on "sshd did not start". So treat a failed bind as + # a lost race and try the next free port up, unless the caller pinned + # one with --port. + local tries=0 from=2222 + while [ "$tries" -lt 5 ]; do + tries=$((tries + 1)) + [ -n "$PORT_PINNED" ] || PORT="$(find_free_port "$from")" || { + echo "ERROR: no free port at or above $from" >&2; exit 3; } + write_sshd_config "$sftp_server" + log "starting throwaway sshd on 127.0.0.1:$PORT" + # Run it as root so any AllowUsers target can log in; it is bound to + # loopback and accepts a single ephemeral key. -E keeps its log, which + # is the only place an auth refusal is ever explained. + if "${SUDO[@]}" "$sshd_bin" -f "$SCRATCH/sshd_config" \ + -E "$SCRATCH/sshd.log" && wait_for_sshd_pid; then + log "sshd pid $SSHD_PID (port $PORT)" + return 0 + fi + # Reap anything that did start before we move the port: otherwise it + # survives teardown, because SSHD_PID is still empty. + "${SUDO[@]}" pkill -f "sshd -f $SCRATCH/sshd_config" 2>/dev/null || true + [ -z "$PORT_PINNED" ] || break + from=$((PORT + 1)) + done + + echo "ERROR: sshd did not start" >&2 + dump_sshd_log + exit 3 +} + +# The config is rewritten per attempt because the Port line changes. +write_sshd_config() { + cat > "$SCRATCH/sshd_config" <&2 + sed 's/^/E: /' "$SCRATCH/sshd.log" >&2 +} + +# Assemble the -o list. Ours first, caller's last so theirs wins. +build_opts() { + local opts="reconnect,ServerAliveInterval=15,ServerAliveCountMax=3" + if [ -z "$HOST" ]; then + # Loopback: a throwaway host key we just generated is not worth + # recording, and would collide on the next run. + opts="$opts,IdentityFile=$SCRATCH/id,IdentitiesOnly=yes" + opts="$opts,StrictHostKeyChecking=no,UserKnownHostsFile=/dev/null" + fi + [ "$NO_CACHE" != 0 ] && opts="$opts,cache=no" + [ -n "$WORKAROUND" ] && opts="$opts,workaround=$WORKAROUND" + [ -n "$EXTRA_OPTS" ] && opts="$opts,$EXTRA_OPTS" + echo "$opts" +} + +mount_sshfs() { + local spec opts + opts="$(build_opts)" + + if [ -z "$HOST" ]; then + start_local_sshd + REMOTE_DIR="${REMOTE_DIR:-$SCRATCH/backing}" + as_invoker mkdir -p "$REMOTE_DIR" + spec="${REMOTE_USER:-$INVOKER_USER}@127.0.0.1:$REMOTE_DIR" + else + [ -n "$REMOTE_DIR" ] || { + echo "ERROR: --host requires --remote-dir" >&2; exit 2; } + spec="${REMOTE_USER:-$INVOKER_USER}@$HOST:$REMOTE_DIR" + fi + + [ -d "$MNT" ] || MNT_CREATED=1 + "${SUDO[@]}" mkdir -p "$MNT" + # Only re-own a directory we made. An explicit --mount-point may be + # someone's existing directory, and teardown has no way to put its + # ownership back. + if [ "$MNT_CREATED" = 1 ]; then + "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$MNT" + fi + + # Remote mode never called start_local_sshd, so PORT may still be + # empty: that is plain ssh, which means 22. + log "sshfs -p ${PORT:=22} -o $opts $spec $MNT" + if ! as_invoker sshfs -p "$PORT" -o "$opts" "$spec" "$MNT"; then + echo "ERROR: sshfs mount failed" >&2 + dump_sshd_log + exit 3 + fi + + # An sshfs mount can appear in the mount table before the first + # request round-trips. Touch it once so a failure surfaces here rather + # than inside the wrapped command. + as_invoker test -d "$MNT" || { echo "ERROR: mount not usable" >&2; exit 3; } + "${SUDO[@]}" mount | grep -F "$MNT" | sed 's/^/I: /' +} + +mount_sshfs + +RUN_HOME="$HOME" +if [ "$SET_HOME" = 1 ]; then + RUN_HOME="$MNT/home" + as_invoker mkdir -p "$RUN_HOME" +fi + +if [ "$INVOKER_UID" != "$(id -u)" ]; then + log "running (as uid=$INVOKER_UID): $*" + # Literal sudo, not "${SUDO[@]}" -- same reason as in as_invoker: the + # array is empty precisely when we are root and need the drop. + sudo -u "$INVOKER_USER" -E \ + env HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" \ + "$@" +else + log "running: $*" + HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" "$@" +fi From daa06ec80debf072ba76e5b7957a8ca00cabca7a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 00:46:24 +0000 Subject: [PATCH 2/4] Put sshfs in the CI matrix, with two cells rather than four The backend was on-demand only, so nothing watched it. Add it as a matrix row: `sshfs (loopback) / git-annex test` and `sshfs (loopback) / git testsuite`, taking the matrix from 20 cells to 22. Not four cells. stress-ng and pjdfstest are needs-root, and a FUSE mount belongs to whoever mounted it -- there is no --no-root-squash equivalent the way there is for NFS -- so those two could only ever report on privilege rather than on the filesystem. run-under.sh already refuses that pair with exit 2; the matrix now expresses the same rule as data. Rather than a hand-kept exclusion list, the backend row carries `no-root: true` and `cell_enabled()` in matrix.sh derives the gap from it plus the target's existing `needs-root`. So the reason lives in one place, a future unprivileged backend gets the behaviour for free, and the grid cannot silently grow a cell that measures privilege. Every consumer asks `cell_enabled` instead of assuming backends x targets: - matrix-json.sh omits the pair, so the workflow never schedules it; - gen-readme-matrix.sh prints `n/a` instead of a badge (README regenerated); - update-status.py keeps it out of status.json, so it publishes no permanently-unknown badge; - render-report.py renders an explicit gap rather than an "unknown" badge, which would read as "not measured yet". `sshfs-git-annex` is expected red and is annotated as such on the report page and in GOTCHAS.md: invisible hardlinks (link() succeeds, st_ino differs, nlink=1) break `git annex add`'s post-link verification. `sshfs-git` is the control -- same mount, a suite that never hardlinks into an object store. Its result is not predicted here; the first run measures it, and GOTCHAS.md gets the answer either way. Verified: shellcheck clean and 28/28 bats; matrix-json.sh emits 22 cells with exactly the two sshfs ones; update-status.py's cell set agrees and omits sshfs-stress-ng / sshfs-pjdfstest; and render-report.py, run over a fabricated 22-cell status.json, writes 22 cell badges plus overall, shows two n/a gaps in the sshfs row, and carries the expected-red note on sshfs-git-annex. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK --- .github/matrix.yaml | 16 +++++++++++++++- GOTCHAS.md | 28 ++++++++++++++++++++++++++++ README.md | 1 + bin/ci/gen-readme-matrix.sh | 15 ++++++++++++++- bin/ci/matrix-json.sh | 3 +++ bin/ci/matrix.sh | 28 ++++++++++++++++++++++++++++ bin/ci/render-report.py | 17 ++++++++++++----- bin/ci/update-status.py | 7 +++++++ 8 files changed, 108 insertions(+), 7 deletions(-) diff --git a/.github/matrix.yaml b/.github/matrix.yaml index 8efa5bb..1e11c08 100644 --- a/.github/matrix.yaml +++ b/.github/matrix.yaml @@ -50,7 +50,8 @@ refs: # than a bare GitHub cancellation. # loop-size-mb backing-image size for `eval-under loop --size`. # needs-root the suite is meaningless without privilege, so the -# NFS backend must export no_root_squash for it. +# NFS backend must export no_root_squash for it, and a +# backend marked `no-root` gets no cell for it at all. # pjdfstest is half privileged-vs-unprivileged # assertions and refuses to run otherwise; stress-ng's # chown/mknod stressors need CAP_CHOWN / CAP_MKNOD. @@ -100,6 +101,19 @@ backends: - backend: nfs version: n/a label: NFS (localhost) + + # `no-root` says this backend cannot hand the wrapped suite privilege: + # a FUSE mount belongs to whoever mounted it, and there is no + # --no-root-squash equivalent the way there is for NFS. So this row has + # no cell for a needs-root target (stress-ng, pjdfstest) -- such a cell + # would report on privilege rather than on the filesystem, which is the + # same reason bin/ci/run-under.sh refuses the pair outright. The two + # cells that remain are the ones that found something: git-annex is red + # here by design (invisible hardlinks), git is the control. + - backend: sshfs + version: n/a + label: sshfs (loopback) + no-root: true - backend: loop version: vfat label: Loop vfat diff --git a/GOTCHAS.md b/GOTCHAS.md index 8c02138..16fd682 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -286,6 +286,34 @@ versions. Whatever BeeGFS does differently, it is not breaking git's index, refs, or object plumbing -- so the cause sits in what git-annex layers on top, or in the syscalls the pjdfstest column is flagging. +### `sshfs (loopback) / git-annex test` + +Red by design, and the reason this backend is in the matrix. SFTP's +`ATTRS` carries no inode number and no link count, so sshfs synthesises +`st_ino` per path and reports `nlink=1`. `git annex add` hardlinks +content into `.git/annex/objects` and then verifies the link; the +verification cannot observe a link that the protocol does not express, so +the add fails with `failed to link to annex`. `hardlink=yes` with +`hardlink-same-inode=no` in `fs-capabilities.sh` is the same finding in +ten seconds rather than twenty minutes. + +Not locked-vs-unlocked and not adjusted-vs-plain: what matters is who +does the ingest. `git add` through the annex filter writes a pointer and +succeeds, which is why git-annex's own `Repo Tests v10 unlocked` group is +green while `annex.addunlocked=true` in a plain v10 repo fails. + +`sshfs (loopback) / git testsuite` is the control: the same mount, a suite +that never hardlinks into an object store. + +### `sshfs (loopback) / stress-ng`, `sshfs (loopback) / pjdfstest` -- no cells + +Not red, absent. A FUSE mount belongs to whoever mounted it and there is +no `--no-root-squash` equivalent, so these suites could only report on +privilege, not on the filesystem. `no-root: true` on the backend row in +`.github/matrix.yaml` drops the pair everywhere (workflow, README grid, +badges, report page), and `bin/ci/run-under.sh` refuses it outright with +exit 2 if asked directly. + ### `Loop ext4 / git-annex test` Pre-dates the matrix; ext4 is the control row, so this one *is* a real diff --git a/README.md b/README.md index 3221e0b..5f41ce0 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ itself is both backend- and suite-agnostic: new filesystems drop in as | BeeGFS 7.4.6 | [![BeeGFS 7.4.6 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex) | [![BeeGFS 7.4.6 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git) | [![BeeGFS 7.4.6 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-stress-ng) | [![BeeGFS 7.4.6 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-pjdfstest) | | BeeGFS 8.1.0 | [![BeeGFS 8.1.0 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex) | [![BeeGFS 8.1.0 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git) | [![BeeGFS 8.1.0 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-stress-ng) | [![BeeGFS 8.1.0 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-pjdfstest) | | NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | +| sshfs (loopback) | [![sshfs (loopback) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git-annex.svg)](https://con.github.io/eval-under/#sshfs-git-annex) | [![sshfs (loopback) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git.svg)](https://con.github.io/eval-under/#sshfs-git) | n/a | n/a | | Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | | Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | diff --git a/bin/ci/gen-readme-matrix.sh b/bin/ci/gen-readme-matrix.sh index ed3d5b8..ba8cc8b 100755 --- a/bin/ci/gen-readme-matrix.sh +++ b/bin/ci/gen-readme-matrix.sh @@ -75,6 +75,12 @@ table_md() { IFS='|' read -r backend version label <<< "$cell" printf '| %s |' "$label" for target in "${EVAL_UNDER_TARGETS[@]}"; do + # A pair that is not a cell gets no badge: there is nothing to + # report, and a badge would imply a result we never measure. + if ! cell_enabled "$backend" "$version" "$target"; then + printf ' n/a |' + continue + fi slug="$(cell_slug "$backend" "$version" "$target")" printf ' [![%s / %s](%s)](%s) |' \ "$label" "$(target_label "$target")" \ @@ -103,4 +109,11 @@ if [ "$CHECK" = 1 ]; then fi cp "$new_readme" "$README" -echo "refreshed the README CI matrix ($((${#EVAL_UNDER_BACKENDS[@]} * ${#EVAL_UNDER_TARGETS[@]})) cells)" +cells=0 +for cell in "${EVAL_UNDER_BACKENDS[@]}"; do + IFS='|' read -r backend version _ <<< "$cell" + for target in "${EVAL_UNDER_TARGETS[@]}"; do + cell_enabled "$backend" "$version" "$target" && cells=$((cells + 1)) + done +done +echo "refreshed the README CI matrix ($cells cells)" diff --git a/bin/ci/matrix-json.sh b/bin/ci/matrix-json.sh index 6c1ca13..b711041 100755 --- a/bin/ci/matrix-json.sh +++ b/bin/ci/matrix-json.sh @@ -36,6 +36,9 @@ entries=() for cell in "${EVAL_UNDER_BACKENDS[@]}"; do IFS='|' read -r backend version label <<< "$cell" for target in "${EVAL_UNDER_TARGETS[@]}"; do + # Not every backend x target pair is a cell -- see cell_enabled() + # in matrix.sh. + cell_enabled "$backend" "$version" "$target" || continue entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") done done diff --git a/bin/ci/matrix.sh b/bin/ci/matrix.sh index 9a2dd70..7f02201 100755 --- a/bin/ci/matrix.sh +++ b/bin/ci/matrix.sh @@ -65,6 +65,14 @@ out.append("declare -A _EU_NEEDS_ROOT=(%s)" % " ".join( out.append("declare -A _EU_NEEDS_GA=(%s)" % " ".join( "[%s]=%s" % (q(t["name"]), q(int(bool(t["needs-git-annex"])))) for t in targets)) +def bslug(b): + return b["backend"] if b["version"] == "n/a" else "%s-%s" % (b["backend"], b["version"]) + +# Optional per-backend flag, defaulting to false, so existing rows need +# no edit. +out.append("declare -A _EU_BACKEND_NO_ROOT=(%s)" % " ".join( + "[%s]=%s" % (q(bslug(b)), q(int(bool(b.get("no-root", False))))) for b in backends)) + # Env overrides win, so these are defaults only. out.append(": \"${EVAL_UNDER_REPO_SLUG:=%s}\"" % q(d["repo-slug"])) out.append(": \"${EVAL_UNDER_SRC_DIR:=%s}\"" % q(d["src-dir"])) @@ -131,3 +139,23 @@ target_loop_size_mb() { echo "${_EU_LOOP_MB[$1]:-100}"; } # two. run-under.sh passes --no-root-squash for these. target_needs_root() { [ "${_EU_NEEDS_ROOT[$1]:-0}" = 1 ]; } target_needs_git_annex() { [ "${_EU_NEEDS_GA[$1]:-0}" = 1 ]; } + +# Can this backend hand the wrapped suite privilege at all? Takes a +# backend *slug* (as backend_slug prints it), not a bare backend name. +backend_no_root() { [ "${_EU_BACKEND_NO_ROOT[$1]:-0}" = 1 ]; } + +# Is x a cell the matrix actually defines? +# The grid is deliberately not fully populated: a backend that cannot run +# as root has no cell for a target that needs root, because such a cell +# would measure privilege rather than the filesystem. Every consumer of +# the matrix asks this rather than assuming backends x targets, so the +# workflow, the README grid, the badges and the report page agree on +# which cells exist. +cell_enabled() { + local backend="$1" version="$2" target="$3" + if backend_no_root "$(backend_slug "$backend" "$version")" \ + && target_needs_root "$target"; then + return 1 + fi + return 0 +} diff --git a/bin/ci/render-report.py b/bin/ci/render-report.py index 06da586..00b6091 100755 --- a/bin/ci/render-report.py +++ b/bin/ci/render-report.py @@ -46,6 +46,7 @@ "beegfs-8.1.0-git-annex": "the bug this repo exists to characterise", "nfs-pjdfstest": "NFS chown/setuid divergence (106 of 1280 assertions)", "loop-ext4-git-annex": "pre-existing, predates this harness", + "sshfs-git-annex": "sshfs hardlinks are invisible: link() succeeds, st_ino differs", } STATE = { @@ -119,8 +120,8 @@ def main() -> int: import yaml m = yaml.safe_load(fh) backends = [(b["backend"] if b["version"] == "n/a" else f"{b['backend']}-{b['version']}", - b["label"]) for b in m["backends"]] - targets = [(t["name"], t["label"]) for t in m["targets"]] + b["label"], bool(b.get("no-root"))) for b in m["backends"]] + targets = [(t["name"], t["label"], bool(t["needs-root"])) for t in m["targets"]] npass = sum(1 for c in cells.values() if c.get("conclusion") == "success") total = len(cells) @@ -129,9 +130,15 @@ def main() -> int: args.outdir / "badges" / "overall.svg") rows = [] - for bslug, blabel in backends: + for bslug, blabel, bno_root in backends: tds = [f"{html.escape(blabel)}"] - for tname, tlabel in targets: + for tname, tlabel, tneeds_root in targets: + # Not a cell -- see no-root in matrix.yaml. Rendered as an + # explicit gap rather than an "unknown" badge, which would + # read as "we have not measured this yet". + if bno_root and tneeds_root: + tds.append('n/a') + continue slug = f"{bslug}-{tname}" c = cells.get(slug, {"conclusion": "unknown"}) concl = c.get("conclusion", "unknown") @@ -155,7 +162,7 @@ def main() -> int: tds.append(f'{body}{meta}{why}') rows.append("" + "".join(tds) + "") - head = "".join(f"{html.escape(l)}" for _, l in targets) + head = "".join(f"{html.escape(l)}" for _, l, _ in targets) repo = "con/eval-under" doc = f""" diff --git a/bin/ci/update-status.py b/bin/ci/update-status.py index 40b77b1..9ef11a1 100755 --- a/bin/ci/update-status.py +++ b/bin/ci/update-status.py @@ -57,6 +57,13 @@ def matrix_cells(matrix_file: Path) -> dict[str, dict]: for b in d["backends"]: bslug = b["backend"] if b["version"] == "n/a" else f"{b['backend']}-{b['version']}" for t in d["targets"]: + # Not every backend x target pair is a cell: a backend that + # cannot run as root (no-root in matrix.yaml) has none for a + # needs-root target, since the cell would measure privilege + # rather than the filesystem. Skipping it here keeps it out of + # status.json, so it publishes no permanently-unknown badge. + if b.get("no-root") and t["needs-root"]: + continue slug = f"{bslug}-{t['name']}" cells[slug] = { "backend": b["backend"], From 634efd15139fc3afbd2a8581b7dea7f287a67a2e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 01:23:17 +0000 Subject: [PATCH 3/4] Characterise the sshfs git cell, and stop counting TODOs as failures The `sshfs / git testsuite` cell came back red, and the previous commit had called it "the control: a suite that never hardlinks into an object store". That was wrong, and measuring it turned up a bug in our own reporting. Root cause of the cell: `git clone ` hardlinks each object and then sanity-checks the result, comparing st_mode/st_ino/st_dev/ st_size/st_uid/st_gid against the source (builtin/clone.c). sshfs synthesises st_ino per path, so the check fails and the clone dies with "hardlink different from source". Plain `git clone` of a local path does not work on sshfs at all -- a broader statement than the git-annex one, and the same mechanism. It accounts for t1507-rev-parse-upstream (20), t1013-read-tree-submodule (58) and t0035-safe-bare-repository (2), each of which clones or adds a submodule in setup. A second, independent cause is the absence of unix sockets: t0301-credential-cache (37, "unable to bind ... Operation not permitted") and t0052-simple-ipc (9/9). `-o disable_hardlink` fixes both this and the git-annex failure, which is the counter-intuitive part worth telling a reporter: sshfs then fails link() with EPERM instead of pretending, and every caller here has a copy fallback that only an honest failure reaches. Measured, with an ext4 control: default disable_hardlink ext4 git annex add (locked) ok ok ok git annex add, annex.addunlocked=true FAIL ok ok git clone FAIL ok ok The reporting bug: `not ok N ... # TODO known breakage` is git's test_expect_failure -- a TAP TODO directive that prove counts as an expected result, not a failure. dump-failure-logs.sh counted those, so this cell reported 351 failed assertions where prove saw 166, and the two worst-looking scripts (t1517-outside-repo at 104, t0450-txt-doc-vs-help at 54) are ones prove reports as *ok*. That sends triage after failures that do not exist; it inflated every git cell, vfat included, not just this one. Both the count and the detail listing now exclude the directive, matching prove. GOTCHAS.md gets the mechanisms, the workaround table, a single-script reproduction recipe, and -- deliberately -- the list of small failures that are *not* explained (t0003-attributes 6, t1091 8, t0610 3, t0021 3, ten scripts with one each), so nobody assumes they share a cause. Method: built the pinned git v2.55.0 locally and ran the whole t0*/t1* range under the sshfs backend (174 scripts, 10366 assertions) plus an ext4 control, which passes the five scripts under suspicion. Reproduced the clone failure minimally, read git's check in clone.c, and confirmed link() returns EPERM under disable_hardlink. One hypothesis was tested and rejected rather than left implied: synthesised inode numbers do not collide at 4000 entries, so that is not behind the whole-suite failures. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK --- GOTCHAS.md | 74 ++++++++++++++++++++++++++++++++++++- bin/ci/dump-failure-logs.sh | 13 ++++++- bin/ci/render-report.py | 1 + 3 files changed, 84 insertions(+), 4 deletions(-) diff --git a/GOTCHAS.md b/GOTCHAS.md index 16fd682..25bc27b 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -302,8 +302,78 @@ does the ingest. `git add` through the annex filter writes a pointer and succeeds, which is why git-annex's own `Repo Tests v10 unlocked` group is green while `annex.addunlocked=true` in a plain v10 repo fails. -`sshfs (loopback) / git testsuite` is the control: the same mount, a suite -that never hardlinks into an object store. +`-o disable_hardlink` fixes it, counter-intuitively: sshfs then fails +`link()` with `EPERM` instead of pretending, git-annex falls back to +copying, and the add succeeds. Measured on a loopback mount here: + +| | sshfs default | sshfs `-o disable_hardlink` | ext4 | +| --- | --- | --- | --- | +| `git annex add` (locked) | ok | ok | ok | +| `git annex add`, `annex.addunlocked=true` | **failed to link to annex** | ok | ok | +| `git clone ` | **fatal: hardlink different from source** | ok | ok | + +So a filesystem that advertises hardlinks it cannot express is worse than +one that admits it has none: every caller here has a copy fallback, and +only the honest failure reaches it. Worth suggesting to a reporter as a +mount option before anything else. + +### `sshfs (loopback) / git testsuite` + +Red, and not for the reason first assumed: git's suite *does* hardlink +into an object store. `git clone ` hardlinks each object and +then sanity-checks the result, comparing `st_mode`, `st_ino`, `st_dev`, +`st_size`, `st_uid` and `st_gid` against the source +(`builtin/clone.c`). sshfs synthesises `st_ino` per path, so the +comparison fails and the clone dies: + +``` +$ git clone a b +fatal: hardlink different from source at 'b/.git/objects/a1/dffc7a...' +``` + +**Plain `git clone` of a local path does not work on sshfs at all.** That +is a broader statement than the git-annex one above, and the same +mechanism. `git clone --no-hardlinks` and `git clone file://...` both +work, as does `-o disable_hardlink` per the table above. + +Two established causes account for most of it: + +- **local clone hardlink verification** -- `t1507-rev-parse-upstream` + (20), `t1013-read-tree-submodule` (58), `t0035-safe-bare-repository` + (2). Each does a local clone or adds a submodule (which clones) in + setup, so one failed setup cascades through the script. +- **no unix sockets** -- `t0301-credential-cache` (37), which reports + `fatal: unable to bind to '.../credential/socket': Operation not + permitted`, and `t0052-simple-ipc` (9/9), whose server never comes up. + `unix-socket=no` in the capability profile predicts both. + +Measured locally against an ext4 control, whole suite (174 scripts, +10366 assertions): 166 failed assertions in 24 scripts. The remainder +are small and **not** yet explained -- `t0003-attributes` (6), +`t1091-sparse-checkout-builtin` (8), `t0610-reftable-basics` (3), +`t0021-conversion` (3), and ten scripts with one each. Do not assume +they share a cause with the two above. + +Reproduce a single script rather than the suite: + +```bash +sudo bin/eval-under sshfs --set-home -- \ + env EVAL_UNDER_GIT_TESTS=t1507-rev-parse-upstream.sh bin/ci/target-git.sh +``` + +Two things this cell taught the harness, recorded so they are not +re-learned: + +- `not ok N ... # TODO known breakage` is git's `test_expect_failure`, a + TAP TODO directive that prove counts as an expected result. Counting + those as failures had this cell reporting 351 failed assertions where + prove saw 166, with `t1517-outside-repo` (104) and + `t0450-txt-doc-vs-help` (54) -- both of which prove reports as **ok** + -- looking like the worst offenders. `bin/ci/dump-failure-logs.sh` now + excludes the directive. +- A script that fails only in the full run is not automatically a + concurrency effect. Both scripts above pass in isolation *and* in the + full suite; it was the counting that differed, not the filesystem. ### `sshfs (loopback) / stress-ng`, `sshfs (loopback) / pjdfstest` -- no cells diff --git a/bin/ci/dump-failure-logs.sh b/bin/ci/dump-failure-logs.sh index 6253b18..23b1215 100755 --- a/bin/ci/dump-failure-logs.sh +++ b/bin/ci/dump-failure-logs.sh @@ -74,10 +74,19 @@ if [ "$TARGET" = "git" ]; then echo "=== git testsuite failures ($results) ===" if [ -d "$results" ]; then # Pass 1: which scripts failed, and how badly. + # + # `not ok N ... # TODO known breakage` is git's test_expect_failure: + # a TAP TODO directive, which prove counts as an expected result and + # not as a failure -- a script whose only "not ok" lines are TODOs is + # reported ok by the harness. Counting them here inflated every git + # cell (the sshfs cell read 351 failed assertions where prove saw + # 166, and two scripts prove called ok appeared as the worst + # offenders at 104 and 54), which sends triage after failures that do + # not exist. Exclude the directive, and match prove. names=() counts=() for out in "$results"/*.out; do [ -e "$out" ] || continue - n="$(grep -c '^not ok ' "$out" 2>/dev/null || true)" + n="$(grep '^not ok ' "$out" 2>/dev/null | grep -vc '# TODO' || true)" [ "${n:-0}" -gt 0 ] || continue names+=("$(basename "${out%.out}")") counts+=("$n") @@ -94,7 +103,7 @@ if [ "$TARGET" = "git" ]; then shown=$((shown + 1)) out="$results/${names[$i]}.out" echo "--- ${names[$i]}: ${counts[$i]} failed ---" - grep '^not ok ' "$out" | head -40 || true + grep '^not ok ' "$out" | grep -v '# TODO' | head -40 || true echo " ... last $GIT_DUMP_TAIL_LINES lines of ${names[$i]}.out:" tail -"$GIT_DUMP_TAIL_LINES" "$out" | sed 's/^/ | /' || true echo diff --git a/bin/ci/render-report.py b/bin/ci/render-report.py index 00b6091..eda791b 100755 --- a/bin/ci/render-report.py +++ b/bin/ci/render-report.py @@ -47,6 +47,7 @@ "nfs-pjdfstest": "NFS chown/setuid divergence (106 of 1280 assertions)", "loop-ext4-git-annex": "pre-existing, predates this harness", "sshfs-git-annex": "sshfs hardlinks are invisible: link() succeeds, st_ino differs", + "sshfs-git": "same cause: local git clone verifies its hardlinks by st_ino", } STATE = { From 91bcf2f891662279689bf67aef5bbb75d902e4fd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 02:05:53 +0000 Subject: [PATCH 4/4] GOTCHAS: the git failure count is ~170, not a fixed 166 CI reported 179 failed assertions for the sshfs git cell on this very commit, where the local full-suite run measured 166. Both are post-fix numbers (the 351 was the TODO miscount), so the spread is run-to-run variance on a network filesystem, not a discrepancy to chase. Say so in the entry rather than leaving a precise-looking 166 for someone to read a future 179 against as a regression. Also record that both failures in the workaround table are now confirmed red in CI on the sshfs cells, not just locally: `sshfs / git-annex test` had never reported before this run (my previous push cancelled it mid-flight), so its expected-red was a local measurement until now. Docs only -- test.yaml's paths filter covers bin/**, provision/**, .github/matrix.yaml and itself, so this does not re-run the 22 cells. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK --- GOTCHAS.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/GOTCHAS.md b/GOTCHAS.md index 25bc27b..b501a5d 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -312,6 +312,9 @@ copying, and the add succeeds. Measured on a loopback mount here: | `git annex add`, `annex.addunlocked=true` | **failed to link to annex** | ok | ok | | `git clone ` | **fatal: hardlink different from source** | ok | ok | +Both rows marked failing here are red in CI as well, on the +`sshfs (loopback)` matrix cells, which is what those cells are for. + So a filesystem that advertises hardlinks it cannot express is worse than one that admits it has none: every caller here has a copy fallback, and only the honest failure reaches it. Worth suggesting to a reporter as a @@ -347,9 +350,12 @@ Two established causes account for most of it: permitted`, and `t0052-simple-ipc` (9/9), whose server never comes up. `unix-socket=no` in the capability profile predicts both. -Measured locally against an ext4 control, whole suite (174 scripts, -10366 assertions): 166 failed assertions in 24 scripts. The remainder -are small and **not** yet explained -- `t0003-attributes` (6), +Measured against an ext4 control, whole suite (174 scripts, 10366 +assertions): 166 failed assertions in 24 scripts locally, 179 in the CI +run of the same commit. Treat the count as "of order 170", not exact -- +it moves by ~10 between runs of the same tree, which is itself worth +knowing before reading a delta as a regression. The two causes above +dominate it either way. The remainder are small and **not** yet explained -- `t0003-attributes` (6), `t1091-sparse-checkout-builtin` (8), `t0610-reftable-basics` (3), `t0021-conversion` (3), and ten scripts with one each. Do not assume they share a cause with the two above.