Decide fate of setup.yml (release-tarball builder)

[krystophny]

PR #54 removes main.yml, the umbrella build-everything CI. setup.yml built
the release tarball that main.yml downloaded for prebuilt dependencies, so it
now has no consumer.

Options:

• Keep it as the prebuilt-dependency bundle for the devcontainer image and fresh
  setups, and update it for the relocated layout (external is now
  $CODE/external, outside infra, so the tarball must bundle it). The current
  TODO in the file notes this.
• Retire it as well, if nothing else needs the tarball.

Decide before the next tag: setup.yml triggers on v* tags and is currently
inconsistent with the relocated layout.

[krystophny]

Automated triage pass (Claude), grounded in the repository at 07d67ba. Verify before acting.

Still open and still undecided: setup.yml is unchanged on main and only partially patched on chore/relocate-infra. Below is the evidence for the decision, and the spec to execute it.

Purpose

setup.yml is the only workflow that fires on a v* tag, so whatever it does is what a tag of this repository means. Right now it means "build a machine-specific dependency tree on a GitHub runner and publish it as code-<tag>.tar.gz". PR #54 deletes its sole consumer, so the next tag will either publish a bundle nobody reads or fail in a way nobody notices until a release is needed. The decision matters beyond tidiness: the "prebuilt-dependency bundle" story in option A of this issue does not survive contact with the actual scripts.

Current state

The tarball has exactly one consumer, and PR #54 deletes it. Grepping every repository checked out alongside this one finds three references to itpplasma/code — README.md, .github/workflows/main.yml, and scripts/setup/devcontainer.sh. Only one of those points at a release asset:

.github/workflows/main.yml:76:  wget https://github.com/itpplasma/code/releases/download/v2025.01.02/code-v2025.01.02.tar.gz -O /tmp/code.tar.gz

.github/workflows/main.yml is deleted on origin/chore/relocate-infra in commit 1faaa2d ("Retire umbrella integration CI; document the development model"), 144 lines removed. Nothing in libneo, SIMPLE, NEO-2, NEO-RT, MEPHIT, KAMEL, GORILLA, rabe, vmecpp or the image definitions references the asset. (Other releases/download hits in scripts/setup/ — OpenBLAS, CGAL, HDF5, LFortran — are upstream third-party downloads, unrelated to this asset. This matters for how the acceptance greps below are written.)

The devcontainer never used it. Option A in the issue body proposes keeping the tarball "for the devcontainer image", but images/devcontainer/Dockerfile:13-14 copies and runs scripts/setup/devcontainer.sh, which at scripts/setup/devcontainer.sh:11-14 clones this repo and runs scripts/setup.sh from source at image-build time, then (line 16) mvs the result to /usr/local/src/code_template. .devcontainer/postCreateCommand.sh symlinks .venv and external/fgsl-1.6.0 out of that template. The branch already updated both for the relocated layout: scripts/setup/devcontainer.sh:18-20 on the branch stashes libneo and external inside the template, and postCreateCommand.sh:4-5 splits INFRA=/workspaces/code from CODE=/workspaces. The devcontainer path is complete without a release asset.

The bundle is nearly empty on a GitHub runner. Every heavy dependency script short-circuits on a system copy:

# scripts/setup/openblas.sh:8
if [ -f /usr/include/cblas.h ] || [ -f /usr/include/openblas/cblas.h ]; then
# scripts/setup/gsl.sh:7
if command -v gsl-config &> /dev/null; then
# scripts/setup/fftw.sh:9
if [ -f /usr/include/fftw3.h ] || [ -f /usr/local/include/fftw3.h ]; then
# scripts/setup/netcdf.sh:12
if command -v nf-config &> /dev/null; then

setup.yml itself apt-installs libopenblas-dev, libgsl-dev, libfftw3-dev, libnetcdf-dev, libnetcdff-dev before calling scripts/setup.sh. scripts/setup.sh:13-20 runs openblas, gsl, fgsl, fftw, netcdf, triangle; the workflow adds mfem.sh separately. So the published external/ contains only fgsl-1.6.0, triangle, and mfem-4.7 — the rest is Debian packages the consumer installs anyway.

What it does contain is path-locked. scripts/setup/gsl.sh:15 and scripts/setup/fftw.sh:26 configure with an absolute --prefix; scripts/setup/netcdf.sh:54 bakes -Wl,-rpath,$INSTALL_PREFIX/lib. scripts/setup/venv.sh:12 creates .venv with absolute shebangs, and scripts/setup/libneo.sh:15 does pip install --verbose --no-build-isolation -e ., which records an absolute editable path in site-packages. The old pipeline only worked because both workflows ran at the identical runner path /home/runner/work/code/code. Extracted anywhere else — at the institute, /proj/plasma/CODE/<username> per README.md:40 (README.md:57-59 on the branch) — it is broken.

The branch's partial fix reintroduces the layout the relocation abolished. On chore/relocate-infra, .github/workflows/setup.yml:79-84 carries a TODO(infra-relocation) and then at lines 94-100:

      - name: Build tarball
        run: |
          set -e
          CODE="$(cd .. && pwd)"
          # flat release archive: infra content + the shared external/ at root
          cp -r "$CODE/external" ./external
          tar -czf /tmp/code.tar.gz .

That produces infra/external/, i.e. external back inside infra — exactly what commit a14b19c ("Treat external as workspace-shared at $CODE/external") moved out. The workflow also uses actions/create-release@v1 and actions/upload-release-asset@v1, both archived by GitHub in 2021, plus actions/checkout@v3 (line 18).

Verdict: retire the tarball. Keep tag→Release, because GitHub already attaches an auto-generated source archive to every release at https://github.com/itpplasma/code/archive/refs/tags/<tag>.tar.gz, which is the only artefact that was ever reproducible.

Reproduction / evidence

The decisive argument is the path-lock. Here is a minimal reproduction of the venv half of it that runs in seconds, with no network and no dependency build. This was actually executed; the output below is real.

mkdir -p /tmp/mre/ws/infra && cd /tmp/mre/ws/infra
python3 -m venv .venv
head -1 .venv/bin/pip
#   observed: #!/tmp/mre/ws/infra/.venv/bin/python3
cd /tmp/mre && mv ws ws2
./ws2/infra/.venv/bin/pip --version; echo "exit=$?"
#   observed: bad interpreter: No such file or directory
#             exit=126
#   expected for a relocatable bundle: prints a version, exit=0

scripts/setup/venv.sh:12 is exactly $PYTHON -m venv .venv, so the published tarball's .venv has this property verbatim. The same argument applies to the --prefix-baked GSL/FFTW/NetCDF installs and the editable libneo install; those are visible by inspection at the line references above and need no runner.

The full-fidelity version, if you want it (needs network and a long build; note -b — the flat-layout behaviour only exists on the branch):

mkdir -p /tmp/ws/infra && cd /tmp/ws/infra
git clone -b chore/relocate-infra https://github.com/itpplasma/code.git .
source scripts/setup.sh        # builds ../external and ../libneo (branch layout)
head -1 .venv/bin/pip          # absolute shebang under /tmp/ws/infra
mv /tmp/ws /tmp/ws2
/tmp/ws2/infra/.venv/bin/pip --version   # bad interpreter

Verification recipe for the replacement workflow (no tag burned):

gh workflow run release.yml --ref chore/relocate-infra   # workflow_dispatch retained
gh run watch
# then, after pushing a throwaway tag v0.0.0-test:
gh release view v0.0.0-test --json assets --jq '.assets | length'
#   expected: 0 — no custom asset; the auto source archives are not listed as assets

Behaviour (BDD)

Scenario: Tagging publishes a Release with no prebuilt asset
  Given .github/workflows/setup.yml has been deleted and .github/workflows/release.yml added
  When the tag v0.0.0-test is pushed to itpplasma/code
  Then a GitHub Release named v0.0.0-test exists with auto-generated notes
  And `gh release view v0.0.0-test --json assets --jq '.assets|length'` prints 0
  And `curl -sIL -o /dev/null -w '%{http_code}' https://github.com/itpplasma/code/archive/refs/tags/v0.0.0-test.tar.gz` prints 200
  And no step in .github/workflows/ invokes scripts/setup.sh or scripts/setup/mfem.sh

Scenario: No workflow or script references the retired asset
  Given the repository at the merge of PR #54 plus this change
  When `grep -rn "itpplasma/code/releases\|upload-release-asset\|actions/create-release\|RELEASE_TOKEN\|code\.tar\.gz" .github/ scripts/ images/ .devcontainer/` runs
  Then it prints no matches
  # This pattern is deliberately narrow. A bare "releases/download" would also match
  # scripts/setup/openblas.sh:13, cgal.sh:19, netcdf.sh:33 and lfortran.sh:30, which are
  # upstream third-party downloads and must NOT be removed.

Scenario: A fresh devcontainer still works without any release tarball
  Given the image built from images/devcontainer/Dockerfile at this commit
    via images/devcontainer/cross_build_push.sh
  When the container starts and .devcontainer/postCreateCommand.sh completes
  Then /workspaces/code/.venv resolves to /usr/local/src/code_template/.venv
  And /workspaces/external/fgsl-1.6.0 exists
  And `source /workspaces/code/activate.sh && python -c "import libneo"` exits 0

Scenario: The new release workflow uses no archived or outdated actions
  Given .github/workflows/release.yml
  When `grep -n "actions/create-release\|actions/upload-release-asset\|checkout@v3" .github/workflows/release.yml` runs
  Then it prints no matches
  # Scoped to release.yml on purpose: .github/workflows/mirror.yml:14 also pins
  # actions/checkout@v3. Bumping it is worthwhile but is out of scope for this issue —
  # file it separately rather than letting it block this PR.

Implementation plan

Land on chore/relocate-infra (PR #54) or as a follow-up PR onto it — not onto main, where main.yml still consumes the asset and would break immediately.

1. Delete .github/workflows/setup.yml. This removes the 60-line apt block, the scripts/setup.sh + scripts/setup/mfem.sh invocation, the tar -czf, the two archived actions, the actions/checkout@v3 pin at line 18, and the TODO(infra-relocation) comment at lines 79-84.
2. Add .github/workflows/release.yml — trigger push: tags: ['v*'] plus workflow_dispatch; permissions: contents: write; a single ubuntu-24.04 job with actions/checkout@v4 (fetch-depth: 0) and one step running gh release create "$GITHUB_REF_NAME" --generate-notes with env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}. No apt block, no build, no asset upload. If a curated changelog is wanted instead of auto-notes, add --notes-file CHANGELOG.md.
3. Retire the RELEASE_TOKEN secret. .github/workflows/setup.yml is its only user in the tree — two env: GITHUB_TOKEN: ${{ secrets.RELEASE_TOKEN }} blocks (lines 101 and 111 on main, 111 and 121 on the branch). The built-in GITHUB_TOKEN with contents: write suffices. Remove the secret from repository settings only after step 2 is green.
4. Update README.md — the "Initial setup" section (around line 40 on main, lines 57-59 on chore/relocate-infra) should state that a v* tag produces a source-only Release and that there is no prebuilt-dependency download; fresh setups run scripts/setup.sh, and the fast path is the devcontainer image.
5. Add a line to docs/development-model.md under the existing ## Release model heading (line 75 on the branch): infra tags are reproducibility snapshots of the setup scripts, not binary distributions; prebuilt dependency stacks are the devcontainer image's job. This is the piece that stops option A resurfacing.
6. Tick the setup.yml checklist item in Tracking: adopt the proposed development model #57 and close this issue with a link to the PR.
7. No sibling-repo changes required. Confirmed by grep across the checked-out org repos: only .github/workflows/main.yml (already deleted on the branch) referenced the asset.

If the group instead wants a genuinely reusable binary stack (option A, done properly), that is a different issue and should be filed separately: it requires relocatable installs (RPATH=$ORIGIN, relocatable pkg-config/CMake config files, --copies venvs or no venv in the bundle) and a matrix over compiler/MPI/libc. Do not smuggle it into this one.

Acceptance criteria

• .github/workflows/setup.yml no longer exists in the tree.
• .github/workflows/release.yml exists and is under 30 lines.
• grep -rn "itpplasma/code/releases\|upload-release-asset\|actions/create-release\|RELEASE_TOKEN\|code\.tar\.gz" .github/ scripts/ images/ .devcontainer/ returns nothing. (Verified: this pattern already returns nothing on chore/relocate-infra once setup.yml is excluded, so deleting that file is sufficient.)
• grep -n "checkout@v3" .github/workflows/release.yml returns nothing. (mirror.yml:14 still pins v3; out of scope — track separately.)
• gh workflow run release.yml on the branch completes green in under 2 minutes.
• Pushing throwaway tag v0.0.0-test creates a Release whose asset list is empty, and curl -sIL -o /dev/null -w '%{http_code}' https://github.com/itpplasma/code/archive/refs/tags/v0.0.0-test.tar.gz prints 200. Delete the tag and release afterwards.
• images/devcontainer/cross_build_push.sh produces an image in which .devcontainer/postCreateCommand.sh completes and source /workspaces/code/activate.sh && python -c "import libneo" exits 0.
• pytest tests/ behaves no differently than before the change. This is a no-regression check only — the suite under tests/ never touched the tarball, and it requires built downstream codes, so "no new failures relative to the pre-change baseline on the same machine" is the bar, not "green".
• The setup.yml checklist item in Tracking: adopt the proposed development model #57 is ticked.

References

• actions/create-release and actions/upload-release-asset were archived by GitHub in 2021 (upload-release-asset on 2021-03-04); both repositories are read-only and unmaintained, and still emit the deprecated set-output command. The maintained replacements are softprops/action-gh-release and the built-in gh release create from the GitHub CLI. https://github.com/actions/create-release, https://github.com/actions/upload-release-asset
• GitHub automatically attaches source archives (.tar.gz, .zip) to every release created from a tag — documented under "About releases" in the GitHub Docs. This is why a hand-rolled source tarball is redundant.
• On the general problem of distributing prebuilt HPC dependency stacks and why relocation is the hard part: T. Gamblin, M. LeGendre, M. R. Collette, G. L. Lee, A. Moody, B. R. de Supinski, S. Futral, The Spack Package Manager: Bringing Order to HPC Software Chaos, SC '15, DOI 10.1145/2807591.2807623. The paper documents Spack's use of RPATH and its handling of install prefixes, i.e. the same class of problem that makes a --prefix-baked tar -czf . non-relocatable. (Spack's binary build caches, which add explicit prefix relocation on install, came later than this paper — do not cite the paper for that feature.) EasyBuild addresses the same problem with per-site module trees rather than portable tarballs. If option A is ever revived, it should be built on one of these rather than on tar -czf ..
• No specific literature reference for the workflow wiring itself; that is an implementation detail internal to the repository.

Scope and risk

Single small PR: one file deleted, one ~20-line file added, two docs touched. The main risk is losing the ability to cut a Release if contents: write on GITHUB_TOKEN is restricted at the organization level — check that before removing RELEASE_TOKEN in step 3, and keep the secret for one release cycle as a fallback. Ordering: must land on or after PR #54, since main's main.yml:76 still downloads the asset.

Related: #57 (tracker — this issue is one of its checklist items), #55 (<DEP>_REF/<DEP>_RELEASE standardization; note that #55's <DEP>_BRANCH naming is already stale against docs/development-model.md on the branch, which uses <DEP>_REF and <DEP>_RELEASE).