Release v1.19.3 to main

.github/dependabot.yml

@@ -48,6 +48,34 @@ updates:
       - dependency-name: "*"
         update-types: ["version-update:semver-major", "version-update:semver-minor"]
 
+  # The appliance rootfs base images (#833, #1163). Deliberately asymmetric, and it has to be:
+  # Dependabot reads this file from the DEFAULT branch only, so the entry lives here on `develop`,
+  # while the path it names exists only on `develop-v2`. `target-branch` is the mechanism for
+  # exactly that — Dependabot resolves `directory` against it and opens the PR there. Anyone
+  # applying "the twins are level" mechanically will read this as wrong; moving it to develop-v2
+  # is what makes it stop running, which is how it went unnoticed (CONTRIBUTING.md § The two lanes).
+  #
+  # The stakes here are higher than for build/* above: the appliance's Debian userland has no apt
+  # at runtime, so a base digest bump plus a rebake IS its patch channel. There is no other way for
+  # a fixed openssl or kernel package to reach a flashed box between releases.
+  - package-ecosystem: "docker"
+    directory: "/os/rootfs"
+    target-branch: "develop-v2"
+    schedule:
+      interval: "weekly"
+    groups:
+      docker-appliance:
+        patterns: ["*"]
+    ignore:
+      # Same policy as the docker entry above, and it binds the golang builder tag: a 1.26 -> 1.27
+      # move is a deliberate rebake, not a security update. It does NOT constrain
+      # `debian:trixie-slim` — a single-segment codename tag yields no semver comparison for these
+      # conditions to act on, so what holds the appliance to trixie is the tag written into the
+      # FROM line, not this block. Digest-only bumps, which are the whole point here, carry no
+      # semver change and so pass both entries untouched (see PRs #729 and #1116).
+      - dependency-name: "*"
+        update-types: ["version-update:semver-major", "version-update:semver-minor"]
+
   # Third-party image digests pinned directly in docker-compose.yml (Tari node + wallet,
   # docker-socket-proxy, caddy) — outside build/*, so the `docker` entry above never sees them,
   # and they feed both channels: pulled on the DIY stack, baked into the appliance image (#833).

.github/workflows/ci.yml

@@ -109,6 +109,16 @@ jobs:
       - name: Scan image for CVEs (Trivy)
         # Gate on actionable (fixable) HIGH/CRITICAL only; accepted findings live in .trivyignore.
         # Must stay green before v1.1 images publish (#282).
+        #
+        # cache: false is load-bearing (#1159). The action caches the vulnerability database under
+        # `cache-trivy-$(date +%F)`, and Actions caches are immutable and scoped per ref — so the
+        # first run of the calendar day on a branch freezes that branch's database until midnight.
+        # On 2026-08-20 the identical dashboard image passed on one branch and failed on another 40
+        # seconds apart, because one restored a snapshot written at 00:15 and the other one written
+        # at 13:33. A gate whose answer depends on when its branch last started a run is not a gate.
+        # Measured cost of downloading it every time: 108.49 MiB from mirror.gcr.io/aquasec/trivy-db
+        # in 3.3s, against the 13.9s the ~997 MB cache restore was taking. It also returns ~7.5 GB of
+        # the repo's 10 GB Actions cache budget, which was 92% near-duplicate copies of this one DB.
         uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
         with:
           image-ref: pithead-${{ matrix.service }}:ci
@@ -117,6 +127,7 @@ jobs:
           ignore-unfixed: true
           exit-code: "1"
           trivyignores: .trivyignore
+          cache: "false"
 
   hadolint:
     name: Dockerfile lint (hadolint)

.github/workflows/os-rootfs.yml

@@ -0,0 +1,114 @@
+name: OS rootfs scan
+
+# Build the appliance rootfs (os/rootfs/Dockerfile) and Trivy-scan it (#833). The appliance's
+# Debian userland — podman, netavark, openssl, the kernel — is frozen at bake time and has no
+# apt at runtime, so this scan is the eye on the base OS between releases. Runs on PRs that
+# touch os/ and weekly on a schedule; a red scheduled run in the Actions tab is the alert, the
+# lychee.yml posture. It is its own workflow (not a build-images matrix entry in ci.yml)
+# because the xmrig prebuild makes this the slowest image in the repo — the paths filter keeps
+# it off stack-only PRs.
+#
+# THIS FILE LIVES ON THE DEFAULT BRANCH (`develop`) AND MUST STAY THERE, even though everything it
+# scans is on `develop-v2` (#1162). GitHub fires `schedule:` from the default branch only, so while
+# this file was develop-v2-only its weekly sweep never ran once — 96 runs over 18 days, every one a
+# `pull_request`. Living on `develop` is only half of it: a job on `develop` still checks out
+# `develop`, which has no os/, so the checkout below names the appliance branch explicitly for every
+# trigger that is not a PR. Do not "fix" the asymmetry by moving the file back — the rule and the
+# one-command check for it are in CONTRIBUTING.md § The two lanes.
+on:
+  pull_request:
+    # A PR whose BASE has no os/ cannot be scanned, and a green check named "Build + scan the
+    # appliance rootfs" that scanned nothing is the same lie in a smaller font. `.trivyignore` is in
+    # the paths filter below and lives on both lanes, so without this a PR editing it on `develop` —
+    # #1156 added four mutes that way — would report the appliance clean under a mute it never
+    # applied. Excluding the two os/-less long-lived branches keeps the filter honest while leaving
+    # stacked appliance branches covered. The appliance-side sync PR is what scans a `.trivyignore`
+    # change for real.
+    branches-ignore:
+      - "develop"
+      - "main"
+    paths:
+      - "os/**"
+      - ".github/workflows/os-rootfs.yml"
+      - ".trivyignore"
+  schedule:
+    - cron: "0 6 * * 1" # Mondays 06:00 UTC, after ci.yml's 05:00 image sweep
+  workflow_dispatch:
+
+# Least privilege (#282): read-only, same as ci.yml.
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  rootfs-image:
+    name: Build + scan the appliance rootfs
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          # A PR run scans the PR (the empty string is checkout's default: the triggering ref).
+          # Everything else — the weekly sweep and any manual dispatch — is asking about the
+          # appliance, which lives on develop-v2 whichever branch the trigger fired from. Naming it
+          # here is what makes the sweep real rather than a green skip (#1162). It also means a
+          # dispatch is a true rehearsal of the schedule instead of a different code path.
+          # Written negated on purpose: in an Actions expression `A && B || C` is not a
+          # ternary, it is real boolean logic, and '' is FALSY. `event == 'pull_request' && ''`
+          # would collapse to '' and fall through to 'develop-v2', so every PR would scan
+          # develop-v2 instead of the PR. The truthy value has to be the develop-v2 arm.
+          ref: ${{ github.event_name != 'pull_request' && 'develop-v2' || '' }}
+          persist-credentials: false
+      - name: Check the appliance tree exists
+        # Only a PR may legitimately miss the tree: a PR into `develop` can reach this workflow
+        # through the .trivyignore path filter, and `develop` has no os/. Every other trigger has
+        # just checked out develop-v2 by name, so a miss there means the checkout did not do what
+        # it was asked — and a quiet skip would report a green weekly sweep of nothing, which is
+        # the exact failure #1162 was filed for. Fail instead.
+        id: tree
+        env:
+          EVENT: ${{ github.event_name }}
+        run: |
+          if [ -f os/rootfs/Dockerfile ]; then
+            echo "present=true" >>"$GITHUB_OUTPUT"
+          elif [ "$EVENT" = "pull_request" ]; then
+            echo "present=false" >>"$GITHUB_OUTPUT"
+            echo "os/rootfs/Dockerfile not on this branch; nothing to scan"
+            echo "The appliance rootfs was NOT scanned: os/rootfs/Dockerfile is not on this PR's branch." \
+              >>"$GITHUB_STEP_SUMMARY"
+          else
+            echo "::error::checked out develop-v2 but os/rootfs/Dockerfile is missing — the appliance checkout did not do what it was asked, so this run cannot report a scan it did not do"
+            exit 1
+          fi
+      - name: docker build os/rootfs
+        if: steps.tree.outputs.present == 'true'
+        # BUILD_COMMIT is a build artifact (os/build-image.sh stamps it; the Dockerfile COPYs
+        # it), so stamp it here too. PITHEAD_UPDATER=rauc matches build-image.sh's default so
+        # the scan covers the updater packages; the test args stay empty — release variant.
+        # images/ holds only .keep here: the staged wizard image is a separate scan target
+        # (ci.yml builds and scans the dashboard image directly).
+        run: |
+          git rev-parse HEAD > os/rootfs/BUILD_COMMIT
+          docker build -f os/rootfs/Dockerfile -t pithead-os-rootfs:ci \
+            --build-arg PITHEAD_UPDATER=rauc .
+      - name: Scan rootfs for CVEs (Trivy)
+        if: steps.tree.outputs.present == 'true'
+        # Same gate as ci.yml's image scan: actionable (fixable) HIGH/CRITICAL only; accepted
+        # findings live in .trivyignore with a rationale and how they clear.
+        #
+        # cache: false for the reason spelled out over ci.yml's scan (#1159) — this is the repo's
+        # other trivy call site. It matters more here: the appliance's Debian userland is baked and
+        # has no apt at runtime, so this scan is the only eye on podman, netavark, openssl and the
+        # kernel between releases, and since #1162 that eye is finally open on a schedule.
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+        with:
+          image-ref: pithead-os-rootfs:ci
+          scanners: vuln
+          severity: HIGH,CRITICAL
+          ignore-unfixed: true
+          exit-code: "1"
+          trivyignores: .trivyignore
+          cache: "false"

.github/workflows/pin-watch.yml

@@ -0,0 +1,120 @@
+name: Upstream pin watch
+
+# Weekly upstream-currency report (#1128). It REPORTS and never bumps: a Tari or monerod minor is
+# a data migration to schedule, not a bump to merge (#1129 carries three one-time migrations and a
+# one-way wallet-DB change). RigForge's xmrig-bump.yml opens a build-verified PR instead — same
+# shape, different output, because XMRig is a drop-in binary and its build gate proves the
+# candidate.
+#
+# ONE tracking issue per lane, edited in place. An issue persists, is assignable and milestonable,
+# and lets a human write "held until the bench is free, here's why" in a comment — which a report
+# artifact nobody opens cannot do, and which is the failure mode #1128 was filed about. Two lanes
+# rather than one table because the two trees have different pins and different owners; merging
+# them would mean one lane's rows silently vanishing whenever the other lane could not be read.
+#
+# This file lives on the DEFAULT branch on purpose. The previous pin-watch.yml lived on
+# `develop-v2`, where a `schedule:` can never fire, and so ran exactly zero times — the appliance
+# pins it watched were never checked once. That is the same defect class as #1048 and #1064.
+#
+# Living on the default branch means only `develop`'s tree gets read, and `develop` has no `os/` —
+# so the appliance's own two pins were invisible in a run that looked complete (#1146). The `lane`
+# matrix below fixes that by checking out `develop-v2` explicitly for a second report. It is the
+# same defect as the paragraph above, one level in: the previous watcher never ran, this one ran
+# with a lane missing. Both look identical from the Actions tab.
+on:
+  schedule:
+    - cron: "30 6 * * 1" # Mondays 06:30 UTC, after the image sweeps
+  workflow_dispatch:
+
+# Least privilege (#282): issues.write is the job's one output — it never pushes or publishes.
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  pin-watch:
+    name: Compare upstream component pins against their latest releases (${{ matrix.lane }})
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    strategy:
+      # fail-fast: false so a lane that cannot report does not cancel the lane that can — a missing
+      # report is the thing this watcher exists to make loud, not a reason to lose the other one.
+      fail-fast: false
+      matrix:
+        include:
+          # `ref: ""` is checkout's own default: the ref that triggered the run. On the weekly
+          # schedule that is the default branch, which is what this lane has always read.
+          - lane: product
+            ref: ""
+            title: "Upstream pin currency (weekly report)"
+          # The appliance rootfs COMPILES docker-compose and cosign from `ARG` values, so nothing
+          # else can see them: dependabot's docker ecosystem reads `FROM` lines, and `cosign` is the
+          # verifier the whole signed-update chain rests on. They exist on this lane only.
+          - lane: appliance
+            ref: develop-v2
+            title: "Upstream pin currency — appliance rootfs (weekly report)"
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ matrix.ref }}
+          persist-credentials: false
+      - name: Build the currency report
+        id: report
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -uo pipefail
+          # NOT `set -e`: a non-zero exit here means "one or more lookups could not run", which is
+          # a result to publish, not a reason to skip publishing it. The exit code is carried to
+          # the last step instead, so a watcher that could not do its job says so in the artefact
+          # a human reads AND fails the run.
+          bash scripts/pin-watch.sh >report.md
+          rc=$?
+          # An empty report is itself a failure to report — never publish silence.
+          [ -s report.md ] || { echo "The pin watcher produced no report at all — see the run log." >report.md; rc=1; }
+          echo "rc=$rc" >>"$GITHUB_OUTPUT"
+      - name: Publish it to this lane's tracking issue
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TITLE: ${{ matrix.title }}
+          RC: ${{ steps.report.outputs.rc }}
+        run: |
+          set -Eeuo pipefail
+          body=$(cat report.md)
+          # Exact title match over the open list, NOT `--search`: the search index lags behind
+          # issue creation by minutes, so a search-based dedup files a second issue on the very
+          # next run and then keeps both stale.
+          n=$(gh issue list --state open --limit 200 --json number,title \
+            --jq "map(select(.title == \"$TITLE\")) | .[0].number // empty")
+          # A run that could not do its job replaces the report with a failure notice, which would
+          # otherwise DELETE the "last fully successful check" date — and that date is the only
+          # thing separating "failed once this morning" from "has been dead for six weeks". Carry
+          # the previous one forward. A watcher whose own silence is invisible is the exact defect
+          # this watcher exists to catch, so it must not have it.
+          if [ "$RC" != "0" ] && [ -n "$n" ]; then
+            prev=$(gh issue view "$n" --json body --jq .body | grep -a "^_Last fully successful check:" || true)
+            # A real `if`, not `[ -n "$prev" ] && body=...`. The && form is safe HERE (set -e is
+            # ignored for a non-final command in an AND-OR list, checked rather than assumed), but
+            # it evaluates to 1 when $prev is empty — so it only stays safe while something else
+            # follows it. That is a reordering hazard for one saved line.
+            if [ -n "$prev" ]; then
+              body="$body"$'\n\n'"$prev (this run could not complete)"
+            fi
+          fi
+          if [ -n "$n" ]; then
+            gh issue edit "$n" --body "$body"
+            echo "updated #$n"
+          else
+            gh issue create --title "$TITLE" --label infra --body "$body"
+          fi
+      - name: Fail the run if any lookup could not be made
+        if: steps.report.outputs.rc != '0'
+        run: |
+          echo "::error::one or more upstream lookups could not run — those pins are UNCHECKED, not current"
+          exit 1
+      # Digest-level check (#1137): does this lane's own docker-compose.yml pins still match what
+      # each tag resolves to right now. Separate from the report above on purpose — a version can
+      # be current while its digest has drifted, and that half-done-bump case is this step's job,
+      # not the report's. A mismatch or failed lookup fails this step, which fails the run.
+      - name: Check third-party image pins against their registries
+        run: bash scripts/resolve-pins.sh

.trivyignore

@@ -13,10 +13,23 @@ CVE-2026-45447
 CVE-2026-23949
 CVE-2026-24049
 
-# util-linux family (bsdutils, libmount1, libuuid1, login, mount, util-linux): the fix is a
-# base-distro point release, 2.41.5-0+deb13u1, and no published python:3.11-slim carries it yet —
-# the newest digest still ships 2.41-5. Same shape as CVE-2026-45447 above: `--ignore-unfixed` hid
-# this until Debian published a fix, at which point it reddened every branch at once without any
-# change on our side. Cleared when a base bump brings 2.41.5-0+deb13u1 in; the weekly CVE sweep
-# (#833) re-surfaces it regardless, so this cannot rot silently.
+# util-linux family (bsdutils, libblkid1, liblastlog2-2, libmount1, libsmartcols1, libuuid1, login,
+# mount, util-linux) — FOUR advisories now, all with the same fix and the same clearing condition.
+# The fix is a base-distro point release, 2.41.5-0+deb13u1, and no published python:3.11-slim
+# carries it: `docker pull python:3.11-slim` returns the very digest build/dashboard/Dockerfile
+# already pins, and it still ships 2.41-5. So this cannot be closed by dependabot or by a manual
+# base bump — only by upstream rebuilding the image on the fixed Debian packages. Verified by
+# pulling the tag and reading dpkg, not from the release notes (#1156).
+#
+# Same shape as CVE-2026-45447 above: `--ignore-unfixed` hid these until Debian published a fix, at
+# which point they reddened every branch at once with no change on our side. 53615 arrived that way
+# on 2026-08-19 and the other three on 2026-08-20, between one CI run and the next on a branch
+# nobody had touched.
+#
+# Cleared when a base bump brings 2.41.5-0+deb13u1 in — `docker run --rm python:3.11-slim
+# dpkg-query -W util-linux` is the one-line check, and all four go together. The weekly CVE sweep
+# (#833) re-surfaces them regardless, so this cannot rot silently.
+CVE-2026-53612
+CVE-2026-53613
+CVE-2026-53614
 CVE-2026-53615