test: versioned wire-contract fixtures + drift guard (#351)

tests/README.md

@@ -42,6 +42,9 @@ holds it from off-box: `ssh miner-0 'flock -n -x /var/lock/rig-e2e.lock true ||
 - macOS-specific behaviour (BSD tools, launchd, the mac process control) → [`e2e/macos.sh`](e2e/macos.sh).
 - Something only provable on real hardware (it actually hashes, MSRs really applied, HugePages
   really reserved) → [`e2e-real.sh`](e2e-real.sh).
+- A change to the sister-API feed or the control `/status` wire shape (a new field, a new `status`
+  string) → update the fixture in [`contract/v1/`](contract/v1/) in the same PR; `run.sh`'s
+  `== contract guard ==` section byte-compares against it and fails otherwise (#351).
 
 ## Conventions
 

tests/contract/v1/README.md

@@ -0,0 +1,111 @@
+# Wire contract v1
+
+Plain-JSON fixtures pinning the two wire shapes pithead's fakes consume from a RigForge worker
+(#351). Nothing outside this repo enforced these before — drift was only ever caught by a manual
+live run against a real stack (the confirmed case: pithead's poller didn't know about #320's
+`noop`/`throttled` statuses until someone hit them live).
+
+## What's covered
+
+| Fixture | Pins | Produced by |
+|---|---|---|
+| [`feed.json`](feed.json) | The sister-API feed (`GET :8081/1\|2/summary`) — the XMRig-passthrough envelope plus the full `rigforge` block RigForge adds. | `api_refresh` (`rigforge.sh`) |
+| [`control-status.json`](control-status.json) | The control path's `GET :8082/status` vocabulary — every `status` string the code can emit, one example record per status, the 503 body served before any change has ever landed, and the `age_seconds` every served body carries. | `_control_status` (`rigforge.sh`) for the terminal/`started` record shapes; `do_GET` / `stage_pending` / `_with_age` (`util/control-server.py`) for `pending`, the no-history 503, and `age_seconds` — see below. |
+
+Both are generated by `tests/run.sh` from the **real** code paths (not hand-written), then
+normalized and byte-compared against the committed file on every run — see
+[`tests/run.sh`](../../run.sh), section `== contract guard: wire-shape fixtures (#351) ==`.
+
+`control-status.json` has two producers because it has two servers of truth. `_control_status()` is
+the single choke point `control_apply`/`control_upgrade` write terminal outcomes through — the right
+source for those record shapes, and cheap to call directly (source `rigforge.sh`, call the function,
+read the file it wrote). But it is **not** the thing pithead's poller actually talks to: that's
+`util/control-server.py`'s `GET /status`, which layers three things on top that only exist in the
+Python receiver and are invisible to a guard that only ever calls `_control_status()` directly — the
+503 no-history body (`do_GET`, when `status.json` has never been written), the `pending` status
+(`stage_pending()`, written the instant `POST /apply` accepts a change, before any oneshot has run),
+and a derived `age_seconds` (`_with_age()`) injected into *every* served body, pending or terminal.
+The guard's `== contract guard ==` section spins up the real `util/control-server.py` over HTTP for
+exactly those three (same harness as the `== black-box: the control server (#236) ==` section above
+it) and checks the results against this fixture too — so a change in either file trips it.
+
+## Normalization
+
+A few fields are wall-clock-, release-version-, or server-random-derived, not part of the wire
+*shape*, and would make every fixture regen a false-positive diff. They're replaced with a fixed
+placeholder before compare — everything else (keys, nesting, types, and every enum literal, above
+all `status`) is byte-strict:
+
+| File | Key(s) normalized | Why |
+|---|---|---|
+| `feed.json` | `rigforge.version`, `rigforge.xmrig_version`, `rigforge.xmrig_commit` | Installed-software provenance — bumps on every RigForge release / xmrig pin update, unrelated to the shape of the object they sit in. |
+| `control-status.json` | `applied_at` (every terminal/`started` example), `accepted_at` (the `pending` example) | `date -u` at write/accept time — different on every run by construction. |
+| `control-status.json` | `age_seconds` (every example) | Computed at SERVE time from `applied_at`/`accepted_at` (`_with_age()`) and never persisted to disk — by definition different on every run, same reason as the timestamp it derives from. |
+| `control-status.json` | `change_id` (the `pending` example only) | Server-generated (`os.urandom(8).hex()` in `stage_change()`) — the guard drives this one through a real `POST /apply`, which does not take a caller-supplied id. The seven terminal/`started` examples keep a real, byte-pinned `change_id` because `_control_status()` takes it as an explicit argument the guard controls. |
+
+`change_id` (the seven terminal/`started` examples), `reason`, `backup`, `changed_keys`, `warnings`,
+and `config_meta.revision` are real output from fixed, hand-picked inputs (the guard supplies the
+arguments), so they're fully reproducible and stay byte-pinned — a change to any of them (e.g. a
+reworded `reason` string, or the revision hash algorithm) is a real wire-visible change and should
+fail the guard. The no-history 503 body is likewise byte-pinned in full, not normalized at all — it
+is a fixed literal (`{"status": "no change applied yet"}`) with no volatile field, so any wording
+change is exactly the kind of drift this guard exists to catch.
+
+## Health values pinned by construction (feed.json)
+
+`feed.json`'s `rigforge.health` block is not normalized — every field there is meant to be
+byte-strict. That only works because the guard forces the inputs `_health_json` reads to a fixed,
+hardware-independent state (this file's own `MEMINFO`/`GOVERNOR_FILE`/`RAPL_DIR`/... exports, plus
+`API_CMD` replacing the worker-API curl). `service_active` needed the same treatment: `_health_json`
+calls the REAL `systemctl is-active "$SERVICE_NAME"` (default `xmrig`), so on any Linux box that
+genuinely has an active `xmrig` unit — every fleet rig — that call would return `active` there and
+nowhere else, byte-diffing the fixture only on rigs. The guard overrides `SERVICE_NAME` to a unit
+name no rig will ever have, so `is-active` reports inactive everywhere: a dev Mac (no `systemctl`
+at all), a CI runner (no such unit), and a mining rig (a real `xmrig` unit, but not under this
+fabricated name) all agree. This pins `service_active: false` and, since both are gated on `sa`
+being true in `_health_json`, `clock_pct_of_boost: null` and `throttling: null` as well.
+
+## The status vocabulary
+
+`control-status.json`'s `statuses` array is cross-checked against the UNION of every literal status
+word `rigforge.sh` actually passes to `_control_status` AND every `"status": "<word>"` literal
+`util/control-server.py` itself writes to a file `GET /status` can serve (both read straight off the
+source, `grep` over call sites — not a hand-maintained list, and not just one side of the wire). A
+new status word landing in *either* file can't land without the guard noticing, fixture or not.
+Current vocabulary: `applied`, `failed`, `noop`, `pending`, `rejected`, `rolled_back`, `started`,
+`throttled`. `noop`/`throttled`/`started` are the non-terminal / idempotent additions from #320;
+`pending` (#344) is `stage_pending()`'s own non-terminal status for a change accepted but not yet
+run — it never comes from `_control_status`, only from the receiver itself.
+
+The control-server.py grep intentionally excludes any `"status": "<word>"` on a line also matching
+`_send(202, ...)` — that's the synchronous accept acknowledgment `POST /apply`/`POST /upgrade`
+return (`"status": "accepted"`), a different value from anything a later `GET /status` call returns,
+and not part of this vocabulary.
+
+## The no-history response
+
+Before any change has ever been accepted on a rig (fresh install, or `state/status.json` never
+written), `GET /status` returns `503 Service Unavailable` with the body documented in
+`no_history` — `{"status": "no change applied yet"}`. It is part of what pithead's poller can
+receive on a genuinely quiet rig, so it is pinned the same as every other example, not just an
+implementation detail.
+
+## The change rule
+
+**Changing either wire shape means updating its fixture in the same PR.** The guard fails loudly
+(a diff, not a silent pass) the moment `rigforge.sh` starts emitting something these files don't
+already describe. Regenerate by hand — copy the relevant block out of the guard's failure output,
+`jq -S` it, and paste it in — or run the guard, read the diff, and edit the fixture to match if the
+new shape is intentional.
+
+Bumping the fixture here does **not** by itself update pithead — that's a paired change, filed
+against pithead as its own PR (or done by whoever owns that side), same as any other cross-repo
+contract change. This directory is the RigForge-side half only.
+
+## Versioning
+
+This is `v1`. A backward-incompatible change to either shape (a key removed, a type changed, a
+status renamed rather than added) gets a new `v2/` directory alongside this one rather than an
+in-place edit, so a sibling repo pinned to `v1` keeps working until it moves. A purely additive
+change (a new key, a new status word) stays in `v1` — see `feed.json`'s own history for a real
+example: `config`/`config_meta` (#253/#254) and `control` (#346) all landed as additive keys.

tests/contract/v1/control-status.json

@@ -0,0 +1,121 @@
+{
+  "examples": {
+    "applied": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "2222222222222222",
+      "changed_keys": [
+        "version"
+      ],
+      "reason": "upgraded to v9.9.9",
+      "source": "control",
+      "status": "applied",
+      "warnings": []
+    },
+    "failed": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "5555555555555555",
+      "changed_keys": [
+        "version"
+      ],
+      "reason": "staged upgrade target malformed (want vX.Y.Z)",
+      "source": "control",
+      "status": "failed",
+      "warnings": []
+    },
+    "noop": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "3333333333333333",
+      "changed_keys": [
+        "version"
+      ],
+      "reason": "already on v9.9.9 — nothing to upgrade",
+      "source": "control",
+      "status": "noop",
+      "warnings": []
+    },
+    "pending": {
+      "accepted_at": "NORMALIZED",
+      "age_seconds": "NORMALIZED",
+      "change_id": "NORMALIZED",
+      "status": "pending"
+    },
+    "rejected": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "6666666666666666",
+      "changed_keys": [
+        "DONATION"
+      ],
+      "reason": "rejected merge-failed",
+      "source": "control",
+      "status": "rejected",
+      "warnings": []
+    },
+    "rolled_back": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": "/b",
+      "change_id": "7777777777777777",
+      "changed_keys": [
+        "watchdog",
+        "DONATION"
+      ],
+      "reason": "miner did not return to a live hashrate; rolled back and live",
+      "source": "control",
+      "status": "rolled_back",
+      "warnings": [
+        "thermal protection changed: watchdog"
+      ]
+    },
+    "started": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "1111111111111111",
+      "changed_keys": [
+        "version"
+      ],
+      "reason": null,
+      "source": "control",
+      "status": "started",
+      "warnings": []
+    },
+    "throttled": {
+      "age_seconds": "NORMALIZED",
+      "applied_at": "NORMALIZED",
+      "backup": null,
+      "change_id": "4444444444444444",
+      "changed_keys": [
+        "version"
+      ],
+      "reason": "throttled — too soon since the last upgrade attempt",
+      "source": "control",
+      "status": "throttled",
+      "warnings": []
+    }
+  },
+  "no_history": {
+    "body": {
+      "status": "no change applied yet"
+    },
+    "http_status": 503
+  },
+  "statuses": [
+    "applied",
+    "failed",
+    "noop",
+    "pending",
+    "rejected",
+    "rolled_back",
+    "started",
+    "throttled"
+  ],
+  "version": 1
+}

tests/contract/v1/feed.json

@@ -0,0 +1,94 @@
+{
+  "connection": {
+    "accepted": 42,
+    "failures": 0,
+    "pool": "poolbox.lan:3333",
+    "rejected": 1,
+    "uptime": 93700
+  },
+  "hashrate": {
+    "total": [
+      1234.5,
+      0,
+      0
+    ]
+  },
+  "hugepages": [
+    1248,
+    1248
+  ],
+  "rigforge": {
+    "config": {
+      "DONATION": 1,
+      "autotune": "disabled",
+      "max_temp_c": 85,
+      "pools": [
+        {
+          "url": "h:3333"
+        }
+      ],
+      "watchdog": "enabled",
+      "watchdog_interval_min": 5
+    },
+    "config_meta": {
+      "changed_at": null,
+      "last_change_id": null,
+      "revision": "50c51399833d2a28",
+      "source": null
+    },
+    "control": {
+      "change_id": "7777777777777777",
+      "reason": "miner did not return to a live hashrate; rolled back and live",
+      "status": "rolled_back"
+    },
+    "health": {
+      "clock_pct_of_boost": null,
+      "firmware": {
+        "board": null,
+        "vendor": null
+      },
+      "governor": null,
+      "hugepages_1g": null,
+      "hugepages_total": null,
+      "msr": "none",
+      "ram": {
+        "channels": 0,
+        "modules": 0,
+        "mts": 0,
+        "rated_mts": 0
+      },
+      "service_active": false,
+      "smt": null,
+      "throttling": null,
+      "xmp": null
+    },
+    "power": {
+      "hs_per_watt": null,
+      "watts": null
+    },
+    "tune": {
+      "applied": null,
+      "autotune": {
+        "enabled": false,
+        "next": null,
+        "schedule": null,
+        "target": null
+      },
+      "candidates_tried": null,
+      "last_best_hs": null,
+      "target": null
+    },
+    "version": "NORMALIZED",
+    "watchdog": {
+      "max_temp_c": 85,
+      "mode": "enabled",
+      "resumes_below_c": 80,
+      "strikes": 0,
+      "temp_c": null,
+      "thermal_hold": false
+    },
+    "xmrig_commit": "NORMALIZED",
+    "xmrig_version": "NORMALIZED"
+  },
+  "uptime": 93780
+}