diff --git a/.github/matrix.yaml b/.github/matrix.yaml index 8efa5bb..1e11c08 100644 --- a/.github/matrix.yaml +++ b/.github/matrix.yaml @@ -50,7 +50,8 @@ refs: # than a bare GitHub cancellation. # loop-size-mb backing-image size for `eval-under loop --size`. # needs-root the suite is meaningless without privilege, so the -# NFS backend must export no_root_squash for it. +# NFS backend must export no_root_squash for it, and a +# backend marked `no-root` gets no cell for it at all. # pjdfstest is half privileged-vs-unprivileged # assertions and refuses to run otherwise; stress-ng's # chown/mknod stressors need CAP_CHOWN / CAP_MKNOD. @@ -100,6 +101,19 @@ backends: - backend: nfs version: n/a label: NFS (localhost) + + # `no-root` says this backend cannot hand the wrapped suite privilege: + # a FUSE mount belongs to whoever mounted it, and there is no + # --no-root-squash equivalent the way there is for NFS. So this row has + # no cell for a needs-root target (stress-ng, pjdfstest) -- such a cell + # would report on privilege rather than on the filesystem, which is the + # same reason bin/ci/run-under.sh refuses the pair outright. The two + # cells that remain are the ones that found something: git-annex is red + # here by design (invisible hardlinks), git is the control. + - backend: sshfs + version: n/a + label: sshfs (loopback) + no-root: true - backend: loop version: vfat label: Loop vfat diff --git a/GOTCHAS.md b/GOTCHAS.md index c2e7675..b501a5d 100644 --- a/GOTCHAS.md +++ b/GOTCHAS.md @@ -75,6 +75,117 @@ So `--no-root-squash` (env: `EVAL_UNDER_NFS_NO_ROOT_SQUASH`) exports with user would. The loop and BeeGFS backends already run the wrapped command as root, so the flag is a no-op there. +### sshfs (`bin/eval-under-sshfs`) + +A throwaway sshd is started on the first free port at or above 2222 -- +its own host key, its own `authorized_keys`, its own pid file, all inside +the run's scratch directory -- and a fresh backing directory is +sshfs-mounted back over it. The system sshd is not used and +`~/.ssh/authorized_keys` is never written to. With `--host` it mounts a +real remote instead, using the caller's ssh config. + +| Knob | Value | Why | +| --- | --- | --- | +| Port | first free `>= 2222` | Two runs at once (a reproduction while a suite is going, two CI cells on one runner) must not collide. `--port` pins it. | +| `-o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3` | always | A dropped connection should fail the command, not wedge it forever. | +| `-o cache=no` | only with `--no-cache` | sshfs caches attributes by default, which hides stale-stat behaviour. This turns off sshfs's own cache, not all caching -- the kernel's 1-second attribute timeout still applies, and the stale-size effect above survives it. On is what users actually have. | +| `-o workaround=rename` | only with `--workaround rename` | Makes sshfs emulate rename-over-existing by unlinking first -- non-atomic, which is what SFTP servers without the POSIX-rename extension force. | +| Mount/command user | the invoking user | A FUSE mount belongs to whoever ran `sshfs`; root cannot read it without `allow_other`. Same treatment `eval-under-nfs` gives `root_squash`. | + +**Hardlinks exist but are not observable, and that is the whole story +for git-annex.** `ln a b` succeeds over SFTP, and then `a` and `b` report +*different* inode numbers and `nlink=1` each: + +``` +ext4 name=a ino=1884275 nlink=2 name=b ino=1884275 nlink=2 +sshfs name=a ino=3 nlink=1 name=b ino=4 nlink=1 +``` + +This is not a misconfiguration, and the link is not fake: in the backing +directory on the server both names really do share one inode with +`nlink=2`. What SFTP cannot carry is *identity* -- its attribute record +has neither an inode number nor a link count -- so sshfs synthesises an +`st_ino` per path and reports `nlink=1` for everything. There is no knob +for it: `use_ino` (removed in libfuse 3) only ever passed through inode +numbers that a filesystem supplies, and sshfs has none to supply, so it +would not have helped under FUSE2 either; sshfs 3.7 offers only +`disable_hardlink`, which makes `link()` fail outright. + +Worse than invisible, and worth knowing when a report mentions truncated +files: immediately after writing one name, the *other* name still reads +back with size 0 through the mount -- with `-o cache=no` as well. The +capability probe reports the inode half as +`hardlink-same-inode=no` / `hardlink-nlink=no` while `hardlink=yes` -- +which is exactly why those two checks exist. `hardlink` alone called +sshfs healthy. + +What it costs, measured with git-annex 10.20240129: + +- `git annex add` on a **locked** branch: fine. The file becomes a + symlink into `.git/annex/objects`, and symlinks work. +- **Any unlocked `git annex add`: fails** -- `foo failed to link to + annex`. add hardlinks the content into the annex and then verifies the + link, and the verification cannot succeed on a filesystem where the + link is invisible. This is not limited to an adjusted branch: a plain + v10 repo fails identically with `annex.addunlocked=true` in git + config, set through `git annex config`, or passed as `-c`. +- `git add` through git's own filter (with `annex.largefiles` matching): + **fine** -- the pointer is committed and `git annex fsck` is clean. + So is `git annex unlock` of an already-committed file. +- `git clone` of a local repo: **fails** -- + `fatal: hardlink different from source at '...'`. git's local-clone + path hardlinks objects and runs the same check. + +So the boundary is not locked-vs-unlocked, and not adjusted-vs-plain: +it is **who does the ingest**. git-annex hardlinking content into the +annex fails; git's filter writing a pointer does not. + +That distinction is easy to get backwards from the test suite alone, +because `git annex test` shows `Repo Tests v10 unlocked` **green** and +only `v10 adjusted unlocked branch` red (11 of 12 in its Init Tests +group, once `add` fails). The green group is not evidence that unlocked +repos are fine here -- the suite's unlocked mode ingests with `git add`. +An earlier revision of this file drew exactly that inference and was +wrong. + +It matters for DataLad, which calls `git annex add`: plain v10 unlocked +repos break for it too, not just adjusted ones. + +**`git annex test` wedges partway through, reproducibly.** Both +full-suite runs stopped at the same place -- `Remote Tests / unavailable +remote / removeKey` -- and sat there until killed (15+ minutes on the +second). On ext4 that same test takes **0.02s and passes**, and the whole +suite finishes in 1m21s. + +It is not an I/O hang. While wedged: + +- the mount stays responsive (`ls` returns immediately), +- git-annex holds **no open files on the mount and no sockets**, +- its threads sit in `futex_do_wait` / `ep_poll`, with no child + processes outstanding. + +That is a process waiting on something internal, not one blocked on the +filesystem. Note also that git-annex sets `annex.sshcaching = false` here +on its own, because ssh control sockets need unix sockets and this mount +has none -- so the run is already on a different code path from a normal +one. + +Two caveats before anyone reports this upstream: the same test passes in +seconds when selected on its own with `-p '/unavailable remote/'`, so it +needs the full-suite context; and this was git-annex 10.20240129 from +Ubuntu 24.04, not a daily build. Re-run it through the reproduce +workflow, which installs the daily build from con/git-annex, before +filing anything. + +**Timestamps are quantised to the second.** Five files created back to +back get one or two distinct mtimes, where every other filesystem +measured gives five. Nothing else in the matrix has a clock this coarse, +which makes sshfs the row that exercises git's racy-timestamp handling. + +**No fifos, no unix sockets.** git-annex says so itself at init +("Detected a filesystem without fifo support") and adapts. Worth knowing +before reading it as a failure. + ### BeeGFS (`bin/eval-under-beegfs`) A containerised cluster (`fixtures/beegfs/docker-compose-v{7,8}.yml`) plus @@ -175,6 +286,110 @@ versions. Whatever BeeGFS does differently, it is not breaking git's index, refs, or object plumbing -- so the cause sits in what git-annex layers on top, or in the syscalls the pjdfstest column is flagging. +### `sshfs (loopback) / git-annex test` + +Red by design, and the reason this backend is in the matrix. SFTP's +`ATTRS` carries no inode number and no link count, so sshfs synthesises +`st_ino` per path and reports `nlink=1`. `git annex add` hardlinks +content into `.git/annex/objects` and then verifies the link; the +verification cannot observe a link that the protocol does not express, so +the add fails with `failed to link to annex`. `hardlink=yes` with +`hardlink-same-inode=no` in `fs-capabilities.sh` is the same finding in +ten seconds rather than twenty minutes. + +Not locked-vs-unlocked and not adjusted-vs-plain: what matters is who +does the ingest. `git add` through the annex filter writes a pointer and +succeeds, which is why git-annex's own `Repo Tests v10 unlocked` group is +green while `annex.addunlocked=true` in a plain v10 repo fails. + +`-o disable_hardlink` fixes it, counter-intuitively: sshfs then fails +`link()` with `EPERM` instead of pretending, git-annex falls back to +copying, and the add succeeds. Measured on a loopback mount here: + +| | sshfs default | sshfs `-o disable_hardlink` | ext4 | +| --- | --- | --- | --- | +| `git annex add` (locked) | ok | ok | ok | +| `git annex add`, `annex.addunlocked=true` | **failed to link to annex** | ok | ok | +| `git clone ` | **fatal: hardlink different from source** | ok | ok | + +Both rows marked failing here are red in CI as well, on the +`sshfs (loopback)` matrix cells, which is what those cells are for. + +So a filesystem that advertises hardlinks it cannot express is worse than +one that admits it has none: every caller here has a copy fallback, and +only the honest failure reaches it. Worth suggesting to a reporter as a +mount option before anything else. + +### `sshfs (loopback) / git testsuite` + +Red, and not for the reason first assumed: git's suite *does* hardlink +into an object store. `git clone ` hardlinks each object and +then sanity-checks the result, comparing `st_mode`, `st_ino`, `st_dev`, +`st_size`, `st_uid` and `st_gid` against the source +(`builtin/clone.c`). sshfs synthesises `st_ino` per path, so the +comparison fails and the clone dies: + +``` +$ git clone a b +fatal: hardlink different from source at 'b/.git/objects/a1/dffc7a...' +``` + +**Plain `git clone` of a local path does not work on sshfs at all.** That +is a broader statement than the git-annex one above, and the same +mechanism. `git clone --no-hardlinks` and `git clone file://...` both +work, as does `-o disable_hardlink` per the table above. + +Two established causes account for most of it: + +- **local clone hardlink verification** -- `t1507-rev-parse-upstream` + (20), `t1013-read-tree-submodule` (58), `t0035-safe-bare-repository` + (2). Each does a local clone or adds a submodule (which clones) in + setup, so one failed setup cascades through the script. +- **no unix sockets** -- `t0301-credential-cache` (37), which reports + `fatal: unable to bind to '.../credential/socket': Operation not + permitted`, and `t0052-simple-ipc` (9/9), whose server never comes up. + `unix-socket=no` in the capability profile predicts both. + +Measured against an ext4 control, whole suite (174 scripts, 10366 +assertions): 166 failed assertions in 24 scripts locally, 179 in the CI +run of the same commit. Treat the count as "of order 170", not exact -- +it moves by ~10 between runs of the same tree, which is itself worth +knowing before reading a delta as a regression. The two causes above +dominate it either way. The remainder are small and **not** yet explained -- `t0003-attributes` (6), +`t1091-sparse-checkout-builtin` (8), `t0610-reftable-basics` (3), +`t0021-conversion` (3), and ten scripts with one each. Do not assume +they share a cause with the two above. + +Reproduce a single script rather than the suite: + +```bash +sudo bin/eval-under sshfs --set-home -- \ + env EVAL_UNDER_GIT_TESTS=t1507-rev-parse-upstream.sh bin/ci/target-git.sh +``` + +Two things this cell taught the harness, recorded so they are not +re-learned: + +- `not ok N ... # TODO known breakage` is git's `test_expect_failure`, a + TAP TODO directive that prove counts as an expected result. Counting + those as failures had this cell reporting 351 failed assertions where + prove saw 166, with `t1517-outside-repo` (104) and + `t0450-txt-doc-vs-help` (54) -- both of which prove reports as **ok** + -- looking like the worst offenders. `bin/ci/dump-failure-logs.sh` now + excludes the directive. +- A script that fails only in the full run is not automatically a + concurrency effect. Both scripts above pass in isolation *and* in the + full suite; it was the counting that differed, not the filesystem. + +### `sshfs (loopback) / stress-ng`, `sshfs (loopback) / pjdfstest` -- no cells + +Not red, absent. A FUSE mount belongs to whoever mounted it and there is +no `--no-root-squash` equivalent, so these suites could only report on +privilege, not on the filesystem. `no-root: true` on the backend row in +`.github/matrix.yaml` drops the pair everywhere (workflow, README grid, +badges, report page), and `bin/ci/run-under.sh` refuses it outright with +exit 2 if asked directly. + ### `Loop ext4 / git-annex test` Pre-dates the matrix; ext4 is the control row, so this one *is* a real diff --git a/README.md b/README.md index 998c765..5f41ce0 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ itself is both backend- and suite-agnostic: new filesystems drop in as | BeeGFS 7.4.6 | [![BeeGFS 7.4.6 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex) | [![BeeGFS 7.4.6 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git) | [![BeeGFS 7.4.6 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-stress-ng) | [![BeeGFS 7.4.6 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-pjdfstest) | | BeeGFS 8.1.0 | [![BeeGFS 8.1.0 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex) | [![BeeGFS 8.1.0 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git) | [![BeeGFS 8.1.0 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-stress-ng) | [![BeeGFS 8.1.0 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-pjdfstest) | | NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | +| sshfs (loopback) | [![sshfs (loopback) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git-annex.svg)](https://con.github.io/eval-under/#sshfs-git-annex) | [![sshfs (loopback) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git.svg)](https://con.github.io/eval-under/#sshfs-git) | n/a | n/a | | Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | | Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | @@ -145,6 +146,14 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \ # the fsync-heavy slow path) sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test' +# Under sshfs, with the attribute cache off +sudo bin/eval-under sshfs --no-cache --set-home -- \ + bash -c 'cd "$HOME" && git annex test' + +# ...or against the reporter's own server, with their mount options +sudo bin/eval-under sshfs --host store.example.org --remote-dir /data/scratch \ + --workaround rename --set-home -- git annex fsck + # Skip teardown to poke around after a failure sudo bin/eval-under beegfs --set-home --keep -- some-failing-command @@ -188,6 +197,7 @@ into `bin/eval-under`, bumped with each release tag. | `bin/eval-under` | Dispatcher: routes to `bin/eval-under-` | | `bin/eval-under-beegfs` | BeeGFS backend (containerised cluster + kernel client mount) | | `bin/eval-under-nfs` | NFS backend (localhost loopback export) | +| `bin/eval-under-sshfs` | sshfs backend (throwaway loopback sshd, or a remote you name) | | `bin/eval-under-loop` | Loop-device backend (dd + losetup + mkfs. + mount) | | `fixtures/beegfs/docker-compose-v7.yml` | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host` | | `fixtures/beegfs/docker-compose-v8.yml` | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane) | diff --git a/bin/ci/dump-failure-logs.sh b/bin/ci/dump-failure-logs.sh index 6253b18..23b1215 100755 --- a/bin/ci/dump-failure-logs.sh +++ b/bin/ci/dump-failure-logs.sh @@ -74,10 +74,19 @@ if [ "$TARGET" = "git" ]; then echo "=== git testsuite failures ($results) ===" if [ -d "$results" ]; then # Pass 1: which scripts failed, and how badly. + # + # `not ok N ... # TODO known breakage` is git's test_expect_failure: + # a TAP TODO directive, which prove counts as an expected result and + # not as a failure -- a script whose only "not ok" lines are TODOs is + # reported ok by the harness. Counting them here inflated every git + # cell (the sshfs cell read 351 failed assertions where prove saw + # 166, and two scripts prove called ok appeared as the worst + # offenders at 104 and 54), which sends triage after failures that do + # not exist. Exclude the directive, and match prove. names=() counts=() for out in "$results"/*.out; do [ -e "$out" ] || continue - n="$(grep -c '^not ok ' "$out" 2>/dev/null || true)" + n="$(grep '^not ok ' "$out" 2>/dev/null | grep -vc '# TODO' || true)" [ "${n:-0}" -gt 0 ] || continue names+=("$(basename "${out%.out}")") counts+=("$n") @@ -94,7 +103,7 @@ if [ "$TARGET" = "git" ]; then shown=$((shown + 1)) out="$results/${names[$i]}.out" echo "--- ${names[$i]}: ${counts[$i]} failed ---" - grep '^not ok ' "$out" | head -40 || true + grep '^not ok ' "$out" | grep -v '# TODO' | head -40 || true echo " ... last $GIT_DUMP_TAIL_LINES lines of ${names[$i]}.out:" tail -"$GIT_DUMP_TAIL_LINES" "$out" | sed 's/^/ | /' || true echo diff --git a/bin/ci/gen-readme-matrix.sh b/bin/ci/gen-readme-matrix.sh index ed3d5b8..ba8cc8b 100755 --- a/bin/ci/gen-readme-matrix.sh +++ b/bin/ci/gen-readme-matrix.sh @@ -75,6 +75,12 @@ table_md() { IFS='|' read -r backend version label <<< "$cell" printf '| %s |' "$label" for target in "${EVAL_UNDER_TARGETS[@]}"; do + # A pair that is not a cell gets no badge: there is nothing to + # report, and a badge would imply a result we never measure. + if ! cell_enabled "$backend" "$version" "$target"; then + printf ' n/a |' + continue + fi slug="$(cell_slug "$backend" "$version" "$target")" printf ' [![%s / %s](%s)](%s) |' \ "$label" "$(target_label "$target")" \ @@ -103,4 +109,11 @@ if [ "$CHECK" = 1 ]; then fi cp "$new_readme" "$README" -echo "refreshed the README CI matrix ($((${#EVAL_UNDER_BACKENDS[@]} * ${#EVAL_UNDER_TARGETS[@]})) cells)" +cells=0 +for cell in "${EVAL_UNDER_BACKENDS[@]}"; do + IFS='|' read -r backend version _ <<< "$cell" + for target in "${EVAL_UNDER_TARGETS[@]}"; do + cell_enabled "$backend" "$version" "$target" && cells=$((cells + 1)) + done +done +echo "refreshed the README CI matrix ($cells cells)" diff --git a/bin/ci/install-backend.sh b/bin/ci/install-backend.sh index 6c5ee54..4d76d7a 100755 --- a/bin/ci/install-backend.sh +++ b/bin/ci/install-backend.sh @@ -9,17 +9,18 @@ # usage: # bin/ci/install-backend.sh # -# backend = beegfs | nfs | loop +# backend = beegfs | nfs | loop | sshfs # version = for beegfs: point release (e.g. 7.4.6, 8.1.0) # for loop: filesystem type (e.g. vfat, ext4, xfs, btrfs) # for nfs: literal "n/a" +# for sshfs: literal "n/a" # # Idempotent enough for CI re-runs; not a full package manager. set -euo pipefail export DEBIAN_FRONTEND=noninteractive -BACKEND="${1:?backend required (beegfs|nfs|loop)}" +BACKEND="${1:?backend required (beegfs|nfs|loop|sshfs)}" VERSION="${2:?version required (BeeGFS version | loop fs name | 'n/a' for nfs)}" # Give unattended-upgrades a moment on ubuntu-22.04 runners rather than @@ -72,6 +73,18 @@ install_nfs() { command -v exportfs } +install_sshfs() { + apt_update + # openssh-sftp-server is what actually serves the mount; on Ubuntu it + # is pulled in by openssh-server, but name it so a slimmer image + # cannot leave us without an sftp-server binary. + apt_install sshfs openssh-server openssh-sftp-server + command -v sshfs + # The backend starts its own sshd, so the system one need not run -- + # but its privilege-separation directory must exist. + sudo mkdir -p /run/sshd +} + install_loop() { local pkg case "$VERSION" in @@ -90,5 +103,6 @@ case "$BACKEND" in beegfs) install_beegfs ;; nfs) install_nfs ;; loop) install_loop ;; - *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop)" >&2; exit 1 ;; + sshfs) install_sshfs ;; + *) echo "unknown backend: $BACKEND (expected beegfs|nfs|loop|sshfs)" >&2; exit 1 ;; esac diff --git a/bin/ci/matrix-json.sh b/bin/ci/matrix-json.sh index 6c1ca13..b711041 100755 --- a/bin/ci/matrix-json.sh +++ b/bin/ci/matrix-json.sh @@ -36,6 +36,9 @@ entries=() for cell in "${EVAL_UNDER_BACKENDS[@]}"; do IFS='|' read -r backend version label <<< "$cell" for target in "${EVAL_UNDER_TARGETS[@]}"; do + # Not every backend x target pair is a cell -- see cell_enabled() + # in matrix.sh. + cell_enabled "$backend" "$version" "$target" || continue entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") done done diff --git a/bin/ci/matrix.sh b/bin/ci/matrix.sh index 9a2dd70..7f02201 100755 --- a/bin/ci/matrix.sh +++ b/bin/ci/matrix.sh @@ -65,6 +65,14 @@ out.append("declare -A _EU_NEEDS_ROOT=(%s)" % " ".join( out.append("declare -A _EU_NEEDS_GA=(%s)" % " ".join( "[%s]=%s" % (q(t["name"]), q(int(bool(t["needs-git-annex"])))) for t in targets)) +def bslug(b): + return b["backend"] if b["version"] == "n/a" else "%s-%s" % (b["backend"], b["version"]) + +# Optional per-backend flag, defaulting to false, so existing rows need +# no edit. +out.append("declare -A _EU_BACKEND_NO_ROOT=(%s)" % " ".join( + "[%s]=%s" % (q(bslug(b)), q(int(bool(b.get("no-root", False))))) for b in backends)) + # Env overrides win, so these are defaults only. out.append(": \"${EVAL_UNDER_REPO_SLUG:=%s}\"" % q(d["repo-slug"])) out.append(": \"${EVAL_UNDER_SRC_DIR:=%s}\"" % q(d["src-dir"])) @@ -131,3 +139,23 @@ target_loop_size_mb() { echo "${_EU_LOOP_MB[$1]:-100}"; } # two. run-under.sh passes --no-root-squash for these. target_needs_root() { [ "${_EU_NEEDS_ROOT[$1]:-0}" = 1 ]; } target_needs_git_annex() { [ "${_EU_NEEDS_GA[$1]:-0}" = 1 ]; } + +# Can this backend hand the wrapped suite privilege at all? Takes a +# backend *slug* (as backend_slug prints it), not a bare backend name. +backend_no_root() { [ "${_EU_BACKEND_NO_ROOT[$1]:-0}" = 1 ]; } + +# Is x a cell the matrix actually defines? +# The grid is deliberately not fully populated: a backend that cannot run +# as root has no cell for a target that needs root, because such a cell +# would measure privilege rather than the filesystem. Every consumer of +# the matrix asks this rather than assuming backends x targets, so the +# workflow, the README grid, the badges and the report page agree on +# which cells exist. +cell_enabled() { + local backend="$1" version="$2" target="$3" + if backend_no_root "$(backend_slug "$backend" "$version")" \ + && target_needs_root "$target"; then + return 1 + fi + return 0 +} diff --git a/bin/ci/render-report.py b/bin/ci/render-report.py index 06da586..eda791b 100755 --- a/bin/ci/render-report.py +++ b/bin/ci/render-report.py @@ -46,6 +46,8 @@ "beegfs-8.1.0-git-annex": "the bug this repo exists to characterise", "nfs-pjdfstest": "NFS chown/setuid divergence (106 of 1280 assertions)", "loop-ext4-git-annex": "pre-existing, predates this harness", + "sshfs-git-annex": "sshfs hardlinks are invisible: link() succeeds, st_ino differs", + "sshfs-git": "same cause: local git clone verifies its hardlinks by st_ino", } STATE = { @@ -119,8 +121,8 @@ def main() -> int: import yaml m = yaml.safe_load(fh) backends = [(b["backend"] if b["version"] == "n/a" else f"{b['backend']}-{b['version']}", - b["label"]) for b in m["backends"]] - targets = [(t["name"], t["label"]) for t in m["targets"]] + b["label"], bool(b.get("no-root"))) for b in m["backends"]] + targets = [(t["name"], t["label"], bool(t["needs-root"])) for t in m["targets"]] npass = sum(1 for c in cells.values() if c.get("conclusion") == "success") total = len(cells) @@ -129,9 +131,15 @@ def main() -> int: args.outdir / "badges" / "overall.svg") rows = [] - for bslug, blabel in backends: + for bslug, blabel, bno_root in backends: tds = [f"{html.escape(blabel)}"] - for tname, tlabel in targets: + for tname, tlabel, tneeds_root in targets: + # Not a cell -- see no-root in matrix.yaml. Rendered as an + # explicit gap rather than an "unknown" badge, which would + # read as "we have not measured this yet". + if bno_root and tneeds_root: + tds.append('n/a') + continue slug = f"{bslug}-{tname}" c = cells.get(slug, {"conclusion": "unknown"}) concl = c.get("conclusion", "unknown") @@ -155,7 +163,7 @@ def main() -> int: tds.append(f'{body}{meta}{why}') rows.append("" + "".join(tds) + "") - head = "".join(f"{html.escape(l)}" for _, l in targets) + head = "".join(f"{html.escape(l)}" for _, l, _ in targets) repo = "con/eval-under" doc = f""" diff --git a/bin/ci/run-under.sh b/bin/ci/run-under.sh index 8580891..4c69ecf 100755 --- a/bin/ci/run-under.sh +++ b/bin/ci/run-under.sh @@ -13,10 +13,11 @@ # usage: # bin/ci/run-under.sh [target] # -# backend = beegfs | nfs | loop +# backend = beegfs | nfs | loop | sshfs # version = for beegfs: point release (e.g. 7.4.6, 8.1.0) # for loop: filesystem type (e.g. vfat, ext4) # for nfs: literal "n/a" +# for sshfs: literal "n/a" # target = git-annex (default) | git | stress-ng | pjdfstest # # env overrides: @@ -43,6 +44,20 @@ target_known "$TARGET" || { exit 1 } +# A root-requiring suite under a backend that can only run as the +# invoking user would measure nothing: every privileged syscall it exists +# to test fails for want of privilege, not because of the filesystem. The +# NFS backend has --no-root-squash for this; sshfs has no equivalent, +# because a FUSE mount belongs to whoever mounted it. Refuse the pair up +# front rather than produce a meaningless red result. +if [ "$BACKEND" = sshfs ] && target_needs_root "$TARGET"; then + echo "$TARGET needs root, and the sshfs backend always runs the wrapped" >&2 + echo "command as the invoking user (a FUSE mount belongs to its mounter)." >&2 + echo "There is no --no-root-squash equivalent here, so this combination" >&2 + echo "cannot measure what the target is for. Try the nfs or loop backend." >&2 + exit 2 +fi + # The target scripts re-derive their own defaults from matrix.sh, but an # override handed to us must survive into the wrapped child. export EVAL_UNDER_SRC_DIR @@ -59,6 +74,7 @@ case "$BACKEND" in # need an export that does not squash root, and need to keep # their privileges rather than being dropped to the invoker. target_needs_root "$TARGET" && opts=(--no-root-squash) ;; + sshfs) opts=() ;; *) echo "unknown backend: $BACKEND" >&2; exit 1 ;; esac diff --git a/bin/ci/update-status.py b/bin/ci/update-status.py index 40b77b1..9ef11a1 100755 --- a/bin/ci/update-status.py +++ b/bin/ci/update-status.py @@ -57,6 +57,13 @@ def matrix_cells(matrix_file: Path) -> dict[str, dict]: for b in d["backends"]: bslug = b["backend"] if b["version"] == "n/a" else f"{b['backend']}-{b['version']}" for t in d["targets"]: + # Not every backend x target pair is a cell: a backend that + # cannot run as root (no-root in matrix.yaml) has none for a + # needs-root target, since the cell would measure privilege + # rather than the filesystem. Skipping it here keeps it out of + # status.json, so it publishes no permanently-unknown badge. + if b.get("no-root") and t["needs-root"]: + continue slug = f"{bslug}-{t['name']}" cells[slug] = { "backend": b["backend"], diff --git a/bin/eval-under-sshfs b/bin/eval-under-sshfs new file mode 100755 index 0000000..3c952fa --- /dev/null +++ b/bin/eval-under-sshfs @@ -0,0 +1,506 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# eval-under-sshfs: run a command with TMPDIR / DATALAD_TESTS_TEMP_DIR (and +# optionally HOME) pointing at an sshfs (FUSE-over-SFTP) mount. +# +# Backend of the eval-under framework. +# +# Two modes: +# +# loopback (default) -- brings up a throwaway sshd on a high port, with +# its own host key and its own authorized_keys, and sshfs-mounts a +# fresh backing directory back over it. Nothing outside the scratch +# dir is touched: the user's ~/.ssh is never written to. +# +# remote (--host) -- sshfs-mounts a directory on a host you name, using +# your existing ssh config/agent. For reproducing a reporter's +# actual setup, where the interesting variable is often their +# server's SFTP implementation rather than sshfs itself. + +set -eu + +# Empty MNT means "derive a per-run mountpoint next to the scratch dir" +# (see the MNT_BASE block below); only an explicit --mount-point / +# EVAL_UNDER_MOUNT pins the mount to a fixed path. +MNT="${EVAL_UNDER_MOUNT:-}" +KEEP="${EVAL_UNDER_KEEP:-0}" +SET_HOME="${EVAL_UNDER_HOME_ON_MOUNT:-0}" + +# Loopback-mode knobs. An empty PORT means "pick a free one": two +# eval-under runs on the same machine (or two CI cells on one runner) +# must not fight over a fixed port. +PORT="${EVAL_UNDER_SSHFS_PORT:-}" +# Remote mode: unset means loopback. +HOST="${EVAL_UNDER_SSHFS_HOST:-}" +REMOTE_USER="${EVAL_UNDER_SSHFS_USER:-}" +REMOTE_DIR="${EVAL_UNDER_SSHFS_DIR:-}" + +# sshfs -o options. EVAL_UNDER_SSHFS_OPTS is a comma-separated string +# appended after ours, so a caller can override any default we set. +EXTRA_OPTS="${EVAL_UNDER_SSHFS_OPTS:-}" +NO_CACHE="${EVAL_UNDER_SSHFS_NO_CACHE:-0}" +WORKAROUND="${EVAL_UNDER_SSHFS_WORKAROUND:-}" + +usage() { + cat <<'EOF' +Usage: eval-under-sshfs [OPTIONS] -- CMD [ARGS...] + +Run CMD with TMPDIR / DATALAD_TESTS_TEMP_DIR (and optionally HOME) +pointing at an sshfs mount. + +By default this is a loopback mount: a throwaway sshd is started on a +high port with its own host key and its own authorized_keys file, and a +fresh backing directory is sshfs-mounted back over it. Your ~/.ssh is +not read or written. With --host, mounts a remote directory instead, +using your normal ssh configuration. + +Requires the `sshfs` package (and, for loopback mode, `openssh-server`). +Install with: apt install sshfs openssh-server + +Options (flag / env var / default / purpose): + + --mount-point PATH EVAL_UNDER_MOUNT (next to scratch dir) + Where to mount sshfs on the host. Default: a fresh per-run + directory next to this run's scratch dir, both under $TMPDIR + (/tmp if unset) -- e.g. with TMPDIR=~/.tmp: + + ~/.tmp/eval-under-sshfs-B3UXj.scratch keys, sshd conf, backing + ~/.tmp/eval-under-sshfs-B3UXj.sshfs where this run mounts it + + A fixed default would collide between concurrent runs and hide + which run owned the mount. Pass this option when a stable path is + wanted. + + --set-home EVAL_UNDER_HOME_ON_MOUNT (unset) + Also set HOME=/home for the wrapped command. + + --keep EVAL_UNDER_KEEP (unset) + Skip teardown; leave the mount (and the throwaway sshd) up for + debugging. + + --port N EVAL_UNDER_SSHFS_PORT (first free >=2222) + Port for the throwaway sshd (loopback mode only). Left unset, the + first free port at or above 2222 is used, and a port lost to a + concurrent run between the scan and the bind is retried on the + next one up, so concurrent runs do not collide. + + --host HOST EVAL_UNDER_SSHFS_HOST (unset) + Mount from a remote host instead of starting a local sshd. Uses + your ssh config, keys and agent as-is. Implies that --port refers + to that host's sshd. + + --user USER EVAL_UNDER_SSHFS_USER (invoking user) + Remote username. Requires --host: the throwaway sshd generated for + loopback mode only admits the invoking user, so any other name + there could only ever be refused. + + --remote-dir PATH EVAL_UNDER_SSHFS_DIR (remote: required) + Directory on the remote host to mount. In loopback mode this + defaults to a fresh scratch directory and should be left alone. + + --no-cache EVAL_UNDER_SSHFS_NO_CACHE (unset) + Pass `-o cache=no`. sshfs caches attributes and directory listings + by default, which papers over exactly the stale-stat behaviour a + reporter may be hitting. Turn the cache off to see the raw + protocol semantics. + + --workaround LIST EVAL_UNDER_SSHFS_WORKAROUND (unset) + Pass `-o workaround=LIST` (sshfs's own compatibility switches, + e.g. `rename`, `truncate`, `buflimit`). `rename` is the + interesting one here: it makes sshfs emulate rename-over-existing + by unlinking the destination first, which is what it has to do + against SFTP servers lacking the POSIX-rename extension -- a + non-atomic rename, and precisely the semantics git and git-annex + assume they can rely on. + + --opt OPTS EVAL_UNDER_SSHFS_OPTS (unset) + Extra comma-separated `-o` options, appended last so they win. + May be given more than once. + + -h, --help + Print this help. + +Environment variables set FOR the wrapped command (all backends): + + TMPDIR = + DATALAD_TESTS_TEMP_DIR = + HOME = /home (only if --set-home) + +Exit status: the wrapped command's exit status. Teardown runs on any +exit (unless --keep). + +Examples: + + # git-annex test over loopback sshfs, cache off + sudo bin/eval-under sshfs --no-cache --set-home -- \ + bash -c 'cd "$HOME" && git init t && cd t && git annex init && git annex test' + + # Reproduce non-atomic rename-over-existing + sudo bin/eval-under sshfs --workaround rename --set-home -- some-command + + # Against a reporter's own server + sudo bin/eval-under sshfs --host store.example.org --remote-dir /data/scratch \ + --set-home -- git annex fsck +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --keep) KEEP=1; shift ;; + --mount-point) MNT="$2"; shift 2 ;; + --set-home) SET_HOME=1; shift ;; + --port) PORT="$2"; shift 2 ;; + --host) HOST="$2"; shift 2 ;; + --user) REMOTE_USER="$2"; shift 2 ;; + --remote-dir) REMOTE_DIR="$2"; shift 2 ;; + --no-cache) NO_CACHE=1; shift ;; + --workaround) WORKAROUND="$2"; shift 2 ;; + --opt) EXTRA_OPTS="${EXTRA_OPTS:+$EXTRA_OPTS,}$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ $# -gt 0 ] || { echo "no command given" >&2; usage >&2; exit 2; } + + +# Did the caller pin the port? Decided before start_local_sshd starts +# reassigning PORT while hunting for a free one. +PORT_PINNED="${PORT:+1}" + +if [ "$(id -u)" -ne 0 ]; then + command -v sudo >/dev/null || { + echo "must run as root (or have sudo available)" >&2; exit 2; } + SUDO=(sudo) +else + SUDO=() +fi + +# The FUSE mount belongs to whoever runs sshfs, and the wrapped command +# has to be able to read it. Both therefore run as the invoking user -- +# the same treatment bin/eval-under-nfs gives root_squash. +if [ -n "${SUDO_UID:-}" ]; then + INVOKER_UID="$SUDO_UID"; INVOKER_GID="${SUDO_GID:-$SUDO_UID}" + INVOKER_USER="${SUDO_USER:-$INVOKER_UID}" +else + INVOKER_UID="$(id -u)"; INVOKER_GID="$(id -g)" + INVOKER_USER="$(id -un)" +fi + +# In loopback mode the generated sshd config has AllowUsers , so +# a different --user could only ever be refused -- with nothing but +# "Connection reset by peer" to explain it. +if [ -n "$REMOTE_USER" ] && [ -z "$HOST" ]; then + echo "--user needs --host: the throwaway sshd only accepts $INVOKER_USER" >&2 + exit 2 +fi + +# Dropping privileges needs `sudo` itself, so this deliberately does NOT +# go through ${SUDO[@]}: that array is EMPTY when we are already root, +# which is exactly the case that needs the drop. Under +# `sudo eval-under sshfs ...` -- the documented invocation, and the one +# bin/ci/run-under.sh uses -- id -u is 0 while INVOKER_UID is the real +# user, so "${SUDO[@]}" -u would expand to a bare `-u`. bin/eval-under-nfs +# spells out `sudo -u` for the same reason. +as_invoker() { + if [ "$INVOKER_UID" = "$(id -u)" ]; then + "$@" + else + sudo -u "$INVOKER_USER" "$@" + fi +} + +# Dropping privileges is only possible with sudo present, whether or not +# we needed it to become root in the first place. +if [ "$INVOKER_UID" != "$(id -u)" ]; then + command -v sudo >/dev/null || { + echo "ERROR: need sudo to run as the invoking user ($INVOKER_USER)" >&2 + exit 2 + } +fi + +log() { printf '\nI: %s\n' "$*"; } + +command -v sshfs >/dev/null 2>&1 || { + echo "ERROR: sshfs not installed. Install with: apt install sshfs" >&2 + exit 3 +} + +# The scratch dir (throwaway host key, sshd config, authorized_keys and +# the backing directory that gets re-exported) and the mountpoint derive +# from one mktemp -u base, so a run leaves a self-describing pair of +# siblings under $TMPDIR -- the convention every backend here follows +# (see "Adding a new backend" in the README). +MNT_BASE="$(mktemp -u "${TMPDIR:-/tmp}/eval-under-sshfs-XXXXX")" +SCRATCH="$MNT_BASE.scratch" +MNT="${MNT:-$MNT_BASE.sshfs}" +mkdir -p "$SCRATCH" +# 755, not mktemp's 700: sshd reads its config and authorized_keys from +# here as root while the mount and the wrapped command run as the +# invoking user. The mode is not enough on its own -- root created the +# directory, so the invoker also has to own it or every as_invoker write +# into it (the keys, authorized_keys, the backing dir) is EACCES. +chmod 755 "$SCRATCH" +chown "$INVOKER_UID:$INVOKER_GID" "$SCRATCH" +SSHD_PID="" + +# Whether teardown created the mountpoint and may therefore remove it. +MNT_CREATED=0 + +teardown() { + local rc=$? + set +e + if [ "$KEEP" = 1 ]; then + echo "I: --keep set; leaving mount up (exit=$rc)" + echo "I: mount: $MNT" + echo "I: scratch: $SCRATCH" + [ -n "$SSHD_PID" ] && echo "I: sshd pid: $SSHD_PID (kill it yourself)" + return "$rc" + fi + log "teardown" + # Kill the wrapped command first. On a signal it is still running, and + # it must not survive the filesystem it is working on. + pkill -P $$ 2>/dev/null + if command -v fuser >/dev/null 2>&1; then + "${SUDO[@]}" fuser -k -M "$MNT" 2>/dev/null + fi + # fusermount as the mount's owner; fall back to a lazy root umount. + as_invoker fusermount3 -u "$MNT" 2>/dev/null \ + || as_invoker fusermount -u "$MNT" 2>/dev/null \ + || "${SUDO[@]}" umount -l "$MNT" 2>/dev/null + # Kill the whole process group: the daemon forks a child per + # connection, and killing only the listener leaves those behind + # holding the port. + [ -n "$SSHD_PID" ] && { "${SUDO[@]}" kill -- "-$SSHD_PID" 2>/dev/null \ + || "${SUDO[@]}" kill "$SSHD_PID" 2>/dev/null; } + # Only remove the mountpoint if this run made it: an explicit + # --mount-point may well be a directory that predates us. + if [ "$MNT_CREATED" = 1 ]; then + "${SUDO[@]}" rm -rf "$MNT" 2>/dev/null + fi + "${SUDO[@]}" rm -rf "$SCRATCH" 2>/dev/null + return "$rc" +} +# Teardown on a signal too, not just a clean exit: otherwise a SIGTERM +# unmounts and rm -rf's the filesystem out from under a wrapped command +# that is still running, and the orphan keeps writing into a deleted +# inode. on_signal drops the EXIT trap so teardown runs exactly once. +on_signal() { + trap - EXIT + teardown + exit 130 +} +trap teardown EXIT +trap on_signal INT TERM HUP + +# First free TCP port at or above $1 on loopback. `ss` if we have it, +# otherwise a bash /dev/tcp connect probe: a refused connection means +# nothing is listening. +find_free_port() { + local p="$1" limit=$(( $1 + 200 )) + while [ "$p" -lt "$limit" ]; do + if command -v ss >/dev/null 2>&1; then + ss -ltnH "sport = :$p" 2>/dev/null | grep -q . || { echo "$p"; return 0; } + else + if ! (exec 3<>"/dev/tcp/127.0.0.1/$p") 2>/dev/null; then + echo "$p"; return 0 + fi + fi + p=$(( p + 1 )) + done + return 1 +} + +find_sftp_server() { + local p + for p in /usr/lib/openssh/sftp-server \ + /usr/libexec/openssh/sftp-server \ + /usr/lib/ssh/sftp-server \ + /usr/libexec/sftp-server; do + [ -x "$p" ] && { echo "$p"; return 0; } + done + return 1 +} + +# Bring up an sshd that exists only for this run: own port, own host key, +# own authorized_keys, own pid file. Deliberately NOT the system sshd -- +# we would otherwise have to append to the user's authorized_keys and +# hope teardown removes it again. +start_local_sshd() { + local sftp_server sshd_bin + sshd_bin="$(command -v sshd || echo /usr/sbin/sshd)" + [ -x "$sshd_bin" ] || { + echo "ERROR: sshd not found. Install with: apt install openssh-server" >&2 + exit 3 + } + sftp_server="$(find_sftp_server)" || { + echo "ERROR: no sftp-server binary found (openssh-sftp-server?)" >&2 + exit 3 + } + + as_invoker ssh-keygen -t ed25519 -N '' -q -f "$SCRATCH/id" -C eval-under-sshfs + as_invoker ssh-keygen -t ed25519 -N '' -q -f "$SCRATCH/hostkey" -C eval-under-sshfs-host + as_invoker cp "$SCRATCH/id.pub" "$SCRATCH/authorized_keys" + chmod 600 "$SCRATCH/authorized_keys" + + # sshd refuses to start without its privilege-separation directory, + # which is absent in minimal containers (it is created at boot by the + # packaging, not by the package install). + "${SUDO[@]}" mkdir -p /run/sshd + + # Finding a free port and binding it are two steps, and a concurrent + # run can take the port in between -- with four runs on one machine, + # three used to die on "sshd did not start". So treat a failed bind as + # a lost race and try the next free port up, unless the caller pinned + # one with --port. + local tries=0 from=2222 + while [ "$tries" -lt 5 ]; do + tries=$((tries + 1)) + [ -n "$PORT_PINNED" ] || PORT="$(find_free_port "$from")" || { + echo "ERROR: no free port at or above $from" >&2; exit 3; } + write_sshd_config "$sftp_server" + log "starting throwaway sshd on 127.0.0.1:$PORT" + # Run it as root so any AllowUsers target can log in; it is bound to + # loopback and accepts a single ephemeral key. -E keeps its log, which + # is the only place an auth refusal is ever explained. + if "${SUDO[@]}" "$sshd_bin" -f "$SCRATCH/sshd_config" \ + -E "$SCRATCH/sshd.log" && wait_for_sshd_pid; then + log "sshd pid $SSHD_PID (port $PORT)" + return 0 + fi + # Reap anything that did start before we move the port: otherwise it + # survives teardown, because SSHD_PID is still empty. + "${SUDO[@]}" pkill -f "sshd -f $SCRATCH/sshd_config" 2>/dev/null || true + [ -z "$PORT_PINNED" ] || break + from=$((PORT + 1)) + done + + echo "ERROR: sshd did not start" >&2 + dump_sshd_log + exit 3 +} + +# The config is rewritten per attempt because the Port line changes. +write_sshd_config() { + cat > "$SCRATCH/sshd_config" <&2 + sed 's/^/E: /' "$SCRATCH/sshd.log" >&2 +} + +# Assemble the -o list. Ours first, caller's last so theirs wins. +build_opts() { + local opts="reconnect,ServerAliveInterval=15,ServerAliveCountMax=3" + if [ -z "$HOST" ]; then + # Loopback: a throwaway host key we just generated is not worth + # recording, and would collide on the next run. + opts="$opts,IdentityFile=$SCRATCH/id,IdentitiesOnly=yes" + opts="$opts,StrictHostKeyChecking=no,UserKnownHostsFile=/dev/null" + fi + [ "$NO_CACHE" != 0 ] && opts="$opts,cache=no" + [ -n "$WORKAROUND" ] && opts="$opts,workaround=$WORKAROUND" + [ -n "$EXTRA_OPTS" ] && opts="$opts,$EXTRA_OPTS" + echo "$opts" +} + +mount_sshfs() { + local spec opts + opts="$(build_opts)" + + if [ -z "$HOST" ]; then + start_local_sshd + REMOTE_DIR="${REMOTE_DIR:-$SCRATCH/backing}" + as_invoker mkdir -p "$REMOTE_DIR" + spec="${REMOTE_USER:-$INVOKER_USER}@127.0.0.1:$REMOTE_DIR" + else + [ -n "$REMOTE_DIR" ] || { + echo "ERROR: --host requires --remote-dir" >&2; exit 2; } + spec="${REMOTE_USER:-$INVOKER_USER}@$HOST:$REMOTE_DIR" + fi + + [ -d "$MNT" ] || MNT_CREATED=1 + "${SUDO[@]}" mkdir -p "$MNT" + # Only re-own a directory we made. An explicit --mount-point may be + # someone's existing directory, and teardown has no way to put its + # ownership back. + if [ "$MNT_CREATED" = 1 ]; then + "${SUDO[@]}" chown "$INVOKER_UID:$INVOKER_GID" "$MNT" + fi + + # Remote mode never called start_local_sshd, so PORT may still be + # empty: that is plain ssh, which means 22. + log "sshfs -p ${PORT:=22} -o $opts $spec $MNT" + if ! as_invoker sshfs -p "$PORT" -o "$opts" "$spec" "$MNT"; then + echo "ERROR: sshfs mount failed" >&2 + dump_sshd_log + exit 3 + fi + + # An sshfs mount can appear in the mount table before the first + # request round-trips. Touch it once so a failure surfaces here rather + # than inside the wrapped command. + as_invoker test -d "$MNT" || { echo "ERROR: mount not usable" >&2; exit 3; } + "${SUDO[@]}" mount | grep -F "$MNT" | sed 's/^/I: /' +} + +mount_sshfs + +RUN_HOME="$HOME" +if [ "$SET_HOME" = 1 ]; then + RUN_HOME="$MNT/home" + as_invoker mkdir -p "$RUN_HOME" +fi + +if [ "$INVOKER_UID" != "$(id -u)" ]; then + log "running (as uid=$INVOKER_UID): $*" + # Literal sudo, not "${SUDO[@]}" -- same reason as in as_invoker: the + # array is empty precisely when we are root and need the drop. + sudo -u "$INVOKER_USER" -E \ + env HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" \ + "$@" +else + log "running: $*" + HOME="$RUN_HOME" TMPDIR="$MNT" DATALAD_TESTS_TEMP_DIR="$MNT" "$@" +fi