CI: htmldoc artifacts and package

[hdiethelm]

This PR adds:

• Artifacts for htmldoc
• A published package for htmldocs Removed
  • Github doesn't support generic packages but containers can be used with oras
  • The package is published for 2.9 / master and any tag
  • Download with a persistent link: oras pull ghcr.io/linuxcnc/linuxcnc/doc-html:master
• Allow sid builds to fail and continue building other packages

If needed for the 2.9 branch, I can backport this.

The discussion started in: #4119

As much as I understand the github docs, only members are allowed to write packages. So PR's should not be able to create a package, even if one removes the if() to only run this stage always.

For testing, I run this stage in my github account: https://github.com/hdiethelm/linuxcnc-fork/actions/runs/27276506335
The package is here: https://github.com/hdiethelm/linuxcnc-fork
And can be downloaded with: oras pull ghcr.io/hdiethelm/linuxcnc/doc-html:ci_doc_build_test

@BsAtHome
Do you thing this does the job?
I will create a commit that removes the if() and see what happens. Then I will revert it again.

[hdiethelm]

And again, the CI broke because sid is broken. Might be we should make the sid package build allow to fail?

I tested it:

apt install python3-opencv
Solving dependencies... Error!  
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

Unsatisfied dependencies:
 python3-opencv : Depends: libopencv-calib3d410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-contrib410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-core410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-dnn410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-features2d410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-flann410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-highgui410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-imgcodecs410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-imgproc410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-ml410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-objdetect410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-photo410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-shape410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-stitching410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-video410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-videoio410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-viz410 (>= 4.10.0+dfsg) but it is not going to be installed
Error: Unable to satisfy dependencies. Reached two conflicting assignments:
   1. libadios2-mpi-core-2.11:amd64 is selected for install because:
      1. python3-opencv:amd64=4.10.0+dfsg-7+b2 is selected for install
      2. python3-opencv:amd64 Depends libopencv-viz410 (>= 4.10.0+dfsg)
      3. libopencv-viz410:amd64 Depends libvtk9.5 (>= 9.5.2+dfsg4)
      4. libvtk9.5:amd64 Depends libadios2-mpi-c++-2.11 (>= 2.11.0+dfsg1)
      5. libadios2-mpi-c++-2.11:amd64 Depends libadios2-mpi-core-2.11 (>= 2.11.0+dfsg1)
   2. libadios2-mpi-core-2.11:amd64 Depends libadios2-mpi-plugins (= 2.11.0+dfsg1-7+b1)
      but none of the choices are installable:
      [no choices]

I guess it will be fixed soon in sid.

[BsAtHome]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

[BsAtHome]

And, yes, a soft-fail on Debian:sid may be appropriate.
It is a rare occasion that sid breaks, but it is a blocker.

[hdiethelm]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

It is a debian package, depending how it is set up, apt-get install oras is all it needs.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

The only issue there is, that artifacts are linked to CI runs. So for an update you would have to download the file manually. Even wget doesn't work to download the file when you use right-click copy link due to you have to be logged in.
You can try it right now, the doc artifact is already enabled: https://github.com/LinuxCNC/linuxcnc/actions/runs/27279180672

Is this good enough?

Otherwise what I found so far:
https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f
I can try that and give you a command that should work. But you will need an access token.

[BsAtHome]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

It is a debian package, depending how it is set up, apt-get install oras is all it needs.

That is a problem, right there... We can't install or sudo on the webserver.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

Is this good enough?

Yes, that was what I was thinking about. Don't know if it works. That's why I was inquiring ;-)

The problem was that the htmldocs run did not produce any artifacts, so we had no chance of testing in any direction.

Otherwise what I found so far: https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f I can try that and give you a command that should work. But you will need an access token.

That looks like a way. Generating a limited access token, just to get the file, may be a possibility.

[hdiethelm]

And, yes, a soft-fail on Debian:sid may be appropriate. It is a rare occasion that sid breaks, but it is a blocker.

With c323d8e, the other packages are built, even if sid fails. But sid is shown red.
I can move the continue-on-error: to all steps, then it will show green. But I think red is fine, it failed and needs investigation.

[hdiethelm]

Otherwise what I found so far: https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f I can try that and give you a command that should work. But you will need an access token.

That looks like a way. Generating a limited access token, just to get the file, may be a possibility.

I found something that works:
https://docs.github.com/en/rest/actions/artifacts?apiVersion=2026-03-10

Token:

Get url's of all artifacts that match the branch and are named linuxcnc-doc:

TOKEN="YourToken"
BRANCH="ci_doc_build"
NAME="linuxcnc-doc"
curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "X-GitHub-Api-Version: 2026-03-10" \
  "https://api.github.com/repos/linuxcnc/linuxcnc/actions/artifacts?per_page=100&name=${NAME}"  | \
  jq ".artifacts[] | select(.workflow_run.head_branch==\"${BRANCH}\") | .archive_download_url"

Download the zip:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "X-GitHub-Api-Version: 2026-03-10" \
  https://api.github.com/repos/LinuxCNC/linuxcnc/actions/artifacts/7537361103/zip -o linuxcnc-doc.zip

Combined:

TOKEN="YourToken"
BRANCH=ci_doc_build
NAME="linuxcnc-doc"
DL_URL=$(curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${TOKEN}" \
-H "X-GitHub-Api-Version: 2026-03-10" \
"https://api.github.com/repos/linuxcnc/linuxcnc/actions/artifacts?per_page=100&name=${NAME}" | \
jq ".artifacts[] | select(.workflow_run.head_branch==\"${BRANCH}\") | .archive_download_url" | head -n1)

#DL_URL has quotes, remove them
DL_URL=${DL_URL//\"/}

curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${TOKEN}" \
-H "X-GitHub-Api-Version: 2026-03-10" \
"$DL_URL" -o linuxcnc-doc.zip

Looks like the links are sorted by date, so I can just take the first one.

Does that work for you? You need curl and jq Took some time to get all together, all i found was to download releases but we need artifacts from CI build.

As soon as this branch is merged, you can change BRANCH=ci_doc_build to BRANCH=master

Edit: Using ?name= is more efficient than filtering for name. Also, there is pagination, i set it to 100. So after 100 other builds without any master build, it will fail.

[hdiethelm]

I removed the commit creating packages.
Should be fine to merge if you are ok with it.

[BsAtHome]

That looks very good that we might be almost there to try.

@andypugh This might be able to restore the devel docs on the webserver. Can you check on the webserver if it has jq installed (try run jq --version)? There should be curl by default, I think.
If jq is there, then we have a "simple" way to poll github for new devel html content directly built from CI (via a cron job). If jq is unavailable, then we need to look at what alternatives can parse json and are available on the webserver (python, php, lua, perl,...).

[hdiethelm]

There will shure be a way. Otherwhise just copy over the binary of jq or cobble together something in bash / awk to get the right url string.
BTW: Artifacs expire after some time, default 90 days, so it might sometimes fail. It's also still github.
The curl --fail options might help handle errors.

[grandixximo]

This also bites package-indep: today's adios2/sid breakage failed package-indep (debian:sid) and fail-fast cancelled the bookworm/trixie indep builds. Could you apply the same allow_fail handling there? With that I'd close my overlapping #4155.

[andypugh]

Can you check on the webserver if it has jq installed (try run jq --version)?

Sorry, I missed this last night, and don't have the relevant keys on my work laptop. I can check when I get home.

[andypugh]

Can you check on the webserver if it has jq installed (try run jq --version)?

Unfortunately not.

[pdx1-shared-a3-06]$ jq --version
Command 'jq' not found, but can be installed with:
apt install jq
Please ask your administrator.
You will have to enable the component called 'universe'

[BsAtHome]

Please try see if PHP and the json lib are installed. Put this is a file (test.php):

<?php
var_dump(json_decode('{"a":123}'));

and run: php ./test.php

[andypugh]

That looks more promising.

[pdx1-shared-a3-06]$ php ./test.php 
object(stdClass)#1 (1) {
  ["a"]=>
  int(123)
}

[BsAtHome]

Very good. We'll go the PHP way.

@hdiethelm the artifact does not include the html directory. It is the content of the directory. I think the zip-file artifact needs to go one level back and include the containing directory.

[grandixximo]

@BsAtHome sounds good, the PHP route it is. Here is a self-contained PHP fetch-and-deploy script for the webserver. It needs only curl plus the PHP json and zip extensions, no third-party tools and no shelling out.

On the artifact packaging point you raised: this script is agnostic to it. It locates the entrypoint after extraction, so it works whether the zip holds the contents of html/ directly (as it does now) or you repackage it to include the containing directory. It also creates the served html path itself as a symlink, so it does not actually need the wrapping directory. If you still repackage for other consumers it keeps working either way.

How it works:

• Lists the linuxcnc-doc artifacts (paginated), picks the newest live one built from master by a trusted in-repo run, and skips if that id is already deployed (state file), so cron can run it often.
• Two-step download so the token never reaches storage: it resolves archive_download_url with the token but does not follow the redirect, then fetches the signed URL on a clean handle with no Authorization header.
• Publishes atomically: the served path is a symlink swapped over with a single rename(), so the tree is never half-written or momentarily absent. The extracted tree is validated (entrypoint present, plausible file count) before it goes live, and old releases are pruned.

Security notes for review: all curl handles pin to https and verify TLS; zip entry names are audited for traversal before extraction; the download is checked against Content-Length and a size ceiling; the token is read with a strict charset check and never logged.

I have lint-checked it on PHP 7.4 and 8.2 (both clean) and unit-tested the security-critical parts (symlink-safe cleanup, the zip-slip audit, unwrap detection, the atomic swap, prune retention). I have not run it end to end against the live API or webserver, so that part is unverified.

Two things to confirm before wiring it up:

• $branch is master, which is correct once this lands and master CI uploads the artifact. If it is currently built from another branch, that top-of-file knob needs to point at it.
• The webserver must follow symlinks at the docroot (Apache +FollowSymLinks / nginx disable_symlinks off), and the cron user needs write access to the docroot's parent. Both are documented in the header.

Are you already drafting the PHP script, or shall I finish this one? Writing it needs no server access. @andypugh would handle the actual install on the webserver (token, cron entry, FollowSymLinks) since he is the one with the keys.

fetch-devel-docs.php.txt

[BsAtHome]

We also need to find out whether Andy's account on the webserver is allowed to write/change the page serving directory... That would be a real stopper ;-)

Is it really necessary to download 100x100 artifacts? Calling the github 100 times in a burst seems like an attempted denial of service and may be flagged. According to above post, these artifacts are sorted by when they were created/run (newest first).

You may want to use references in your for/foreach loops to reduce the number of copies created of values/instances/arrays (reduces memory footprint).

Keeping 5 releases by default is currently more then 1.5 GB. A bit overkill.

Failing on missing zip extension should not be necessary. We must test the availability of both cURL and zip modules before this can even start to become working. And when the modules are there, then they are there. Besides, we need to see how much memory is available to php-cli because we are handling large files and php is not always the best to keep a low-profile memory footprint.

[grandixximo]

Thanks, all fair. Updated the script:

• Pagination: you are right that walking up to 100 pages was wrong. Since the list is filtered by name and returned newest-first, it now takes the first live in-repo master artifact and stops, fetching another page only if one has no match, capped at 3. In practice that is a single request.
• Releases: dropped the default to keep 2 (the live one plus a single rollback) instead of 5.
• Capabilities: moved the curl/zip/json checks to an up-front preflight that fails clearly before any work, and removed the mid-run zip check.
• Loops: the artifact scan now iterates by reference (with unset after) to avoid copying the arrays.

On memory: the script never loads a file into PHP. curl streams the download straight to a file handle and ZipArchive extracts entry by entry to disk, so memory_limit is not a factor regardless of tree size; only the small JSON metadata is decoded in memory.

Andy's account on the webserver is allowed to write/change the page serving directory...

Agreed, that is the real gate. The cron user needs write on the parent of the served path, since publishing renames a symlink there. If that account cannot write it, none of this works and we would need a different publish path. Worth confirming before going further.

fetch-devel-docs-v2.php.txt

[andypugh]

I do have write access, and have used it to put up a placeholder text.

https://linuxcnc.org/docs/devel/html/index.html

[andypugh]

One question is why the docs disappeared rather than simply not being updated. Is there a risk that the buildbot (if, indeed, it is the buildbot) will delete the docs after each successful build unless we stop it?

Though we can probably prevent that by de-authorising the key:

command="/home/emcboard/bin/rsync-server 'www.linuxcnc.org/docs/*'" ssh-rsa <...elided...> Buildbot doc uploader

[andypugh]

If you are curious, the rsync-server script there does not do the rsync.

#!/usr/bin/python
#
# This script is intended to help provide safe receipt of incoming file
# transfers via rsync.  It should be used in the receiving user's
# .ssh/authorized_keys file as the command associated with the sender's
# public key, like this:
#
#     command="rsync-server www.linuxcnc.org/docs/*" $PUBKEY Buildbot doc uploader
#
# This script examines the command that the sender wanted to run (in
# SSH_ORIGINAL_COMMAND), and does two sanity checks.
# 
#     1. The first two arguments must be "rsync --server".
#
#     2. The last argument must match the glob in $1.
#
# If both those checks pass, it execs the requested command and the file
# transfer goes through.
#

[BsAtHome]

One question is why the docs disappeared rather than simply not being updated. Is there a risk that the buildbot (if, indeed, it is the buildbot) will delete the docs after each successful build unless we stop it?

My guess is that it stopped because the files in question are no longer available at the location they were expected to be. But certainty is only provided by reading/disabling the old sync-code. It seems that by blocking the script/pubkey you can disable the buildbot update?

It broke right after we changed the build layout. That does not seem to be random. Anyway, we could always move the new tree right next to the old directory and update all links, if that would ever be required.

BTW, could you try to run this to see if the some expected extensions are available or we need workarounds for them too:

<?php
$f = false;
foreach (['curl', 'zip', 'json'] as $ext) {
    if (!extension_loaded($ext)) {
        echo "required PHP extension not loaded: $ext\n";
        $f = true;
    }
}
if (!function_exists('symlink')) {
    echo "symlink() is disabled (disable_functions); cannot publish atomically\n";
        $f = true;
}
if(!$f) {
    echo "All fine!\n";
}

[BsAtHome]

Now I'm wondering what happens to the 2.9 branch docs when it gets updated. Do the webserver docs still update in that 2.9 tree automatically?

[BsAtHome]

And, for the record, I hate reviewing LLM generated code. Please install a brain and use it.

[grandixximo]

On #7 I was avoiding parsing ls (shellcheck SC2012 flags ls -1dt), which is why it looks heavier than it needs to. I will trim it down.

[hdiethelm]

Same here, I smell the LLM's from far away, way to much code to solve a simple problem and way to long comments with funny UTF symbols in it. LLM's often just leak the laziness of a developer to find a solution that's simple, short and elegant. In principle I have nothing against LLM's if the code is good quality and de-sloped / reviewed by hand after generating.

The bash script looks already better.

• What I would add is checking the sha256 sum after download. So if something goes wrong in the redirect, we catch it.
• The return value of curl is not checked, if the host returns 404, the zip will not change. But set -e and ^^ will catch that anyway.

Here is the json blob for an artifact:

  {
      "id": 7537361103,
      "node_id": "MDg6QXJ0aWZhY3Q3NTM3MzYxMTAz",
      "name": "linuxcnc-doc",
      "size_in_bytes": 228886335,
      "url": "https://api.github.com/repos/LinuxCNC/linuxcnc/actions/artifacts/7537361103",
      "archive_download_url": "https://api.github.com/repos/LinuxCNC/linuxcnc/actions/artifacts/7537361103/zip",
      "expired": false,
      "digest": "sha256:c0b65691d88f58cf26d7aa5f22b138cf3ef07e8b2fe847f7d28af24b9e6da27e",
      "created_at": "2026-06-10T13:38:07Z",
      "updated_at": "2026-06-10T13:38:07Z",
      "expires_at": "2026-09-08T13:20:02Z",
      "workflow_run": {
        "id": 27279180672,
        "repository_id": 3662905,
        "head_repository_id": 1157775434,
        "head_branch": "ci_doc_build",
        "head_sha": "41d37df8541b619f560083c6ad513c4fe9834252"
      }
    }

[hdiethelm]

@BsAtHome Any reason to have the images executable? Options:

• Just do a chown -r in the CI before creating the artifact.
• The original images are committed with +x. You have a clue why? Someone just did not take care? I can remove the executable flags in git.

edit: I hope that doesn't break anything but having images executable is probably not a good idea. See next commit.

[grandixximo]

Added the sha256 check, the filter now pulls digest alongside the id and url, and after download I verify sha256sum against it (guarded on the field being present, so an older artifact without it does not hard-fail). Worth confirming on the first real run that GitHub's digest is the sha256 of the downloaded zip and not of something pre-rezip, but the blob you posted suggests it is.

On the curl return: curl -fsSL already fails on a 404 (-f) and set -e aborts, so a bad redirect never reaches the unzip. Left as is.

fetch-devel-docs.sh.txt

I am preventing any executable from passing, so permissions have to be fixed for this not to fail.

Edit:
You just fixed the images, thanks

[BsAtHome]

The images were added with exec on them and git faithfully adds them that way. Typical for images from windoze systems and copies from FAT formatted partitions that were mounted without stripping exec.

Simply removing the exec bits in this commit should be fine.

[hdiethelm]

Added the sha256 check, the filter now pulls digest alongside the id and url, and after download I verify sha256sum against it (guarded on the field being present, so an older artifact without it does not hard-fail). Worth confirming on the first real run that GitHub's digest is the sha256 of the downloaded zip and not of something pre-rezip, but the blob you posted suggests it is.

The doc says:

"description": "The SHA256 digest of the artifact. This field will only be populated on artifacts uploaded with upload-artifact v4 or newer. For older versions, this field will be null."

Due to we introduce the artifact here and use v7, this check is not needed. Hard fail is more secure. And fix it if it ever happens, which I don't think it will.

[hdiethelm]

The images were added with exec on them and git faithfully adds them that way. Typical for images from windoze systems and copies from FAT formatted partitions that were mounted without stripping exec.

Simply removing the exec bits in this commit should be fine.

So, it looks good now, no executable images any more.

[grandixximo]

Due to we introduce the artifact here and use v7, this check is not needed

Revised, missing digest aborts, mismatch aborts, no other checks.

fetch-devel-docs.sh.txt

[BsAtHome]

(pedantic) It is customary to add a space after the '<' in $(< "fff") for readability.
On testing the URL... [ false ] && [ true ] || echo xxx does not trigger the echo.
digest is not tested for after extraction.
You removed the shellopt guard in '*' glob. It is not "nice" to compare a glob pattern if it does not expand.

[grandixximo]

Thanks, all applied:

Added the space for <
Fixed id url check with an explicit if, and it now validates digest as well, right after parsing.
Restored the nullglob guard around the cleanup glob.

fetch-devel-docs.sh.txt

[BsAtHome]

The grep -rIl may better be written as grep -rIq? You don't need the output anyway.

BTW, -I triggers on "binary files". When is that triggered? Can HTML files trigger this? I think you may need to set LC_ALL=C.UTF-8 for the grep to prevent tripping on UTF-8 binary sequences if the environment setup is tricky. Can you confirm this?

[grandixximo]

Switched to grep -rIq (dropped the >/dev/null) and set LC_ALL=C so only a NUL byte marks a file binary, never an encoding hiccup. Valid UTF-8 pages are still scanned, -I on my machine only skips real binaries (aka images).

@andypugh could you confirm on the webserver:

grep --version | head -1
printf '<html>caf\xc3\xa9 \xf0\x9f\x98\x80 <?php x ?></html>' > /tmp/u.html
LC_ALL=C grep -rIq -e '<?php' /tmp/u.html && echo "caught (good)" || echo "MISSED"
rm -f /tmp/u.html

Should print caught (good): a UTF-8 page with accents and emoji containing <?php is detected. If it prints MISSED, the grep there handles binary/locale differently and I will adjust.

fetch-devel-docs.sh.txt

[hdiethelm]

Anyway, who uses .zip in the Linux world... Changed the doc artifact to .tar.gz. It's also nicer so no need to use this path expansion trick to have the the html folder inside.

However, there is one tiny downside: If you need this artifact in an other CI job, you need to extract it. If you use the other variant, it arrives extracted.

[andypugh]

I commented in the wrong thread. But would Python be a good choice for the downloading script in the cron job?
I have played around with this using the GitHub copilot and have a script that almost does the right things (and doesn't look hideously obviously AI)

[andypugh]

https://github.com/andypugh/DocsUploader/blob/main/download_ci_artifacts.py

I am not committed to Python, it's just the php felt like a bit of a hack.

[grandixximo]

@hdiethelm the English doc styling bug I mentioned in #4152 turned out to be a master-side issue, now fixed in #4174 (merged). English pages were rendering from build/adoc/en with a stem missing the en/ prefix, so their stylesheet path was one level short (../ instead of ../../) and loaded a nonexistent en/asciidoctor.css.

The re-run did not pick it up: a GitHub "re-run" rebuilds the same PR merge commit from the original trigger (before #4174), it does not recompute the merge against current master. I downloaded the latest tar and English is still ../.

Could you rebase ci_doc_build on current master (or push any commit) so CI does a fresh merge and builds with the fix? After that the tar should be correct. I verified locally that a clean build with #4174 gives ../../ for every English page, with man pages and translations unchanged.

[andypugh]

I tried re-running the jobs to see if the update in #4174 worked, but for whatever reason that didn't have the desired effect.
But I have seen this create an uploadable docs pack, and that's definitely a great starting point.

[andypugh]

I am not clear if this will also create docs for 2.9?

[grandixximo]

Not yet. #4150 is master-only, so only the devel tar is produced. On 2.9 the htmldocs job builds the docs but does not tar or upload them (its only artifacts there are the Debian packages). Covering 2.9 would mean backporting the tar + upload step and running a second cron with BRANCH=2.9 publishing to docs/stable.

Before that though, it is worth pinning down how the buildbot has been updating docs, since the cron is meant to replace it. From this thread the webserver side is just the rsync-server forced-command wrapper; the actual per-branch build and rsync live on @SebKuzminsky's buildbot and have not been described. docs/stable is stale since 15 December (2.9.7-9), so 2.9 may not be auto-updating anyway. The main thing to avoid is the buildbot being re-enabled and an rsync --delete wiping the manually placed (or cron-placed) docs. Could @SebKuzminsky describe how the doc upload currently works per branch?

[grandixximo]

@andypugh about the script:

A short list, split into must-have vs optional in case you want a minimal version:

Essential (or it will not deploy):

• Fetch the right thing: the artifact is named linuxcnc-doc and is a .tar.gz, so match that name and use tarfile, not the htmldocs pattern with zipfile.
• Publish: move the extracted tree into the webroot with an atomic swap. Right now it only extracts to a timestamped dir, nothing gets served.
• Skip if unchanged: record the deployed artifact id and exit early, else cron re-downloads ~220MB every run and never cleans up.
• requests is third-party: confirm it is on the server, or use stdlib urllib.

Nice to have (your call if worth it):

• sha256 check of the download against the artifact digest.
• reject <?php and executables in the extracted tree.
• prune old releases, keeping one for rollback.

[andypugh]

The main thing to avoid is the buildbot being re-enabled and an rsync --delete wiping the manually placed (or cron-placed) docs.

I have already disabled the buildbot's key on the server.

[hdiethelm]

https://github.com/andypugh/DocsUploader/blob/main/download_ci_artifacts.py

I am not committed to Python, it's just the php felt like a bit of a hack.

Still looks like AI, way to much functionality. Also it doesn't check head_repository_id. This is dangerous: When someone creates a PR from his fork branch master to linuxcnc branch master, the script will pull the artifact from his latest PR build, what ever is in there. @grandixximo's AI figured this out, I did not look well enough but my code was also more to show how you can do it in as few lines as possible without AI, so it is slim and no boiler plate... ;-)

I am not clear if this will also create docs for 2.9?

This will: #4176 ;-) Let's see if the CI is happy with it.

[grandixximo]

lets continue the script discussion here

LinuxCNC/wlo#42

Changed to python, tried to slim it down, happy to reshape it however instructed.