diff --git a/examples/nokv-authority-store/README.md b/examples/nokv-authority-store/README.md index 96f601e37a..d33c5a65d6 100644 --- a/examples/nokv-authority-store/README.md +++ b/examples/nokv-authority-store/README.md @@ -17,7 +17,12 @@ This probe intentionally exercises the current Python SDK bridge because it is the available raw byte-CAS seam. It always starts this checkout's reviewed `NoKVJsonLinesTransport` and `nokv_jsonl_helper.py`; it has no fake, skip, or "unverified but successful" CLI path. The helper admits exactly NoKV SDK -`0.11.0` / Python API `1`, and the successful report repeats both values. +`0.11.1` / Python API `1` whose `Client.publish_bytes` names +`expected_workspace_incarnation_id` and whose module exports +`WorkspaceIncarnationMismatch`; the successful report repeats both version +values. A `0.11.0` wheel is refused by the version pin (`invalid_config`), and a +wheel labelled `0.11.1` without that publication surface is refused as +`nokv_sdk_capability_mismatch` before any client is constructed. ## What it proves @@ -25,6 +30,10 @@ Against one **already existing** NoKV workbench, the probe starts three independent helper processes and verifies: - the selected tenant/goal path is initially absent and can be created; +- a publication of the generation-1 envelope fenced on a stale workbench + incarnation is refused typed (`store_identity_mismatch`) before any row or + object exists: the stored generation stays 1 and the workbench identity is + unchanged (every LoopX publication carries the incarnation it was read from); - the stored path generation advances from 1 to 2 under exact generation CAS; - after the generation-2 CAS lands, an injected response loss is reconciled from the durable authority envelope and operation receipt rather than from @@ -42,30 +51,34 @@ If the SDK, helper, workbench, backend, CAS, or independent readback cannot be proved, the process exits nonzero. The normal test suite uses deterministic fakes only to test this sequence and does **not** count as live evidence. -The probe does not prove an atomic expected-incarnation publication fence, -runtime shadow parity, a multi-Agent canary, authority promotion, HA, failover, -restart recovery, capacity, or performance. NoKV generation can restart after -workbench recreation, so the current adapter fails closed through authoritative -post-write readback; preventing the stale-incarnation write itself requires a -future provider primitive. The probe also does not create a workbench. A green -run is Stage 2A single-node storage conformance evidence only. +The probe proves the expected-incarnation publication fence only as far as a +single live owner can show it: a stale fence is refused typed and leaves the +generation untouched. It does not restore or recreate the workbench (NoKV +exposes no client-side retire verb), so the incarnation rotation itself is +covered by NoKV's own executor tests, not by this probe. Success is still +accepted only after authoritative post-write readback in the current +incarnation: the fence removes the stale write, not the readback obligation. +The probe does not prove runtime shadow parity, a multi-Agent canary, authority +promotion, HA, failover, restart recovery, capacity, or performance, and it does +not create a workbench. A green run is Stage 2A single-node storage conformance +evidence only. ## Inputs Use a current NoKV Python environment. Keep the client configuration in an ignored local file; do not commit credentials. The helper accepts three routing kinds and passes each to the matching `RoutingConfig` constructor of the -installed SDK: `etcd` and `static` (the 0.11.0 release wheel) and `seeds` +installed SDK: `etcd` and `static` (the 0.11.x release wheels) and `seeds` (`{"kind": "seeds", "endpoints": ["IP:PORT", ...]}`, the NoKV metadata-runtimes line, which names serving owners directly and drops the etcd constructor). Static routing is valid for a single-node NoKV deployment; etcd is not required by this probe. A routing kind the installed wheel cannot build fails the open handshake with `nokv_sdk_capability_mismatch` before any client -is constructed, so a seeds configuration against a 0.11.0 wheel (or an etcd +is constructed, so a seeds configuration against a 0.11.1 wheel (or an etcd configuration against a metadata-runtimes wheel) is reported as the wrong wheel, not as an outage. The `ready` handshake echoes `nokv_protocol_schema`: the SDK's `WORKSPACE_PROTOCOL_SCHEMA` when the wheel exports one, otherwise -`null` (the 0.11.0 release does not). The following shape is illustrative: +`null` (the 0.11.x releases do not). The following shape is illustrative: ```json { @@ -136,7 +149,7 @@ unfenced, pre-existing, or unreadable state exits nonzero with a compact JSON reason; provider stderr, endpoints, credentials, and raw SDK errors are not copied into that result. A successful JSON report includes `"qualification_scope":"stage_2a_single_node_store_conformance"`, -`"nokv_sdk_version":"0.11.0"`, and `"nokv_api_version":1`. The two version +`"nokv_sdk_version":"0.11.1"`, and `"nokv_api_version":1`. The two version fields are the helper's admission constants: the helper refuses to open a client for any other SDK version or API version, so a successful report implies them, but they are not values read back from the NoKV server. The report is Stage diff --git a/examples/nokv-authority-store/live-qualification.ts b/examples/nokv-authority-store/live-qualification.ts index 8301f67aac..f5fe31031d 100644 --- a/examples/nokv-authority-store/live-qualification.ts +++ b/examples/nokv-authority-store/live-qualification.ts @@ -30,7 +30,7 @@ import { const REPORT_SCHEMA = "loopx_nokv_authority_live_qualification_v0"; export const QUALIFICATION_SCOPE = "stage_2a_single_node_store_conformance"; -export const QUALIFIED_NOKV_SDK_VERSION = "0.11.0"; +export const QUALIFIED_NOKV_SDK_VERSION = "0.11.1"; export const QUALIFIED_NOKV_API_VERSION = 1; const REPOSITORY_HELPER = fileURLToPath( new URL("../../loopx/control_plane/coordination/nokv_jsonl_helper.py", import.meta.url), @@ -266,6 +266,16 @@ function applied( return result; } +/** A 32-hex incarnation that differs from `current` in its first digit. */ +function staleIncarnation(current: string): string { + return `${current.startsWith("0") ? "1" : "0"}${current.slice(1)}`; +} + +/** A fresh lower-layer publication identity; the probe never reuses one. */ +function freshPhysicalIdentity(): string { + return randomUUID().replaceAll("-", ""); +} + async function rawGeneration( transport: NoKVBlobTransport, store: NoKVAuthorityStore, @@ -361,6 +371,47 @@ export async function exerciseQualificationSequence( await rawGeneration(firstTransport, first, 1, "create_generation_failed"); passed("create_generation_one"); + // A write prepared against one workbench incarnation must not land after + // the workbench is restored to another. Send the generation-1 envelope back + // through the raw transport with a fence naming a different incarnation: + // NoKV must refuse it typed, before any row or object exists, and the + // stored generation and the workbench identity must both be unchanged. + const boundIdentity = firstIdentity.status === "available" + ? firstIdentity.store_identity + : fail("workbench_identity_failed", "workbench identity was not available"); + const currentEnvelope = await firstTransport.readBlob(first.workbench, first.path); + expect( + currentEnvelope.status === "loaded" && currentEnvelope.generation === 1, + "stale_incarnation_fence_probe_failed", + "the generation-1 envelope could not be read for the fence probe", + ); + const staleFence = await firstTransport.casPublishBlob({ + workbench: first.workbench, + path: first.path, + expected_generation: 1, + expected_workspace_incarnation_id: staleIncarnation( + boundIdentity.slice(`nokv:${workbench}:`.length), + ), + bytes: currentEnvelope.bytes, + operation_id: freshPhysicalIdentity(), + artifact_revision_id: freshPhysicalIdentity(), + }); + expect( + staleFence.status === "failed" && staleFence.reason_code === "store_identity_mismatch", + "stale_incarnation_fence_not_enforced", + "NoKV did not refuse a publication fenced on a stale workbench incarnation", + ); + passed("stale_incarnation_fence_rejected"); + const identityAfterFence = await first.storeIdentity(); + expect( + identityAfterFence.status === "available" && + identityAfterFence.store_identity === boundIdentity, + "stale_incarnation_fence_probe_failed", + "the workbench incarnation changed during the fence probe", + ); + await rawGeneration(firstTransport, first, 1, "stale_incarnation_fence_wrote"); + passed("stale_incarnation_fence_left_generation_unchanged"); + const advanced = applied( await second.commitAuthority( commit(created.provider_revision, runId, operationIds.advance, 2, "advance"), diff --git a/examples/shared-goal-authority-e2e/README.md b/examples/shared-goal-authority-e2e/README.md index 71e2292798..f72887e37e 100644 --- a/examples/shared-goal-authority-e2e/README.md +++ b/examples/shared-goal-authority-e2e/README.md @@ -35,7 +35,7 @@ suites rather than in the pytest shards. | `s0.file_matrix_twelve_rows` | 0 | store_direct | deterministic | `examples/nokv-shadow-provider/live_e2e.py` reports exactly the twelve known file-provider scenario rows, all true | | `s0.nokv_live_matrix` | 0 | store_direct | env:nokv_legacy | the same twelve rows plus `restored_lineage_fails_closed` are true on a live NoKV stack and file/NoKV outcomes are identical | | `s1.cli_document_decodes_through_ts_store` | 1 | real_cli | deterministic | three CLI writes (`todo add`, `task-lease acquire`, `todo update`) read back through `FileAuthorityStore`: `loadAuthority` loaded at cursor `3`, paged `scanCommitted` yields the three `observation_id`s in order, `readReceipt` finds the first | -| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.0` / API `1`, and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name | +| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.1` / API `1`, the two stale-incarnation fence checks (`stale_incarnation_fence_rejected`, `stale_incarnation_fence_left_generation_unchanged`), and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name | | `s2b.postgresql_conformance_live` | 2b | store_direct | env:postgresql | `postgresql_authority_store.integration.test.ts` under node's TAP reporter: `# pass >= 9`, `# fail 0`, `# skipped 0` | | `s2c1.configure_enable_disable_roundtrip` | 2c1 | real_cli | deterministic | `configure-goal` preview does not write, enable writes, captured observations for a todo and a lease, read-back summary `enabled/file_one_way`, disable writes and later writes neither observe nor touch candidate bytes | | `s2c1.every_writer_family_captures` | 2c1 | real_cli | deterministic | handoff-mode set, todo add/update/complete/supersede/archive-completed, task-lease acquire/renew/transfer each carry `outcome in {captured, replayed, ambiguous_reconciled}`, `primary_writeback_preserved=true`, `provider_to_local_writes=false`, `candidate_read_for_decision=false`; an idempotent re-acquire carries no `authority_shadow`; candidate `cursor == captured count`, operation ids equal observation ids, no time-active lease in the head, head todos equal `todo list` | @@ -86,7 +86,7 @@ TypeScript store read. | `deterministic` | none (needs `node` on `PATH` for the CLI's TypeScript runtime and the read-back probe) | `node_missing` when the probe cannot run | | `env:postgresql` | `LOOPX_TEST_POSTGRES_URL` plus `node_modules/pg` (`npm ci`) | `postgres_url_missing`, `pg_dependency_missing`, `node_missing` | | `env:nokv_legacy` | `NOKV_COORDINATION_LIVE=1` and `NOKV_ETCD`, `NOKV_ETCD_PREFIX`, `NOKV_ROOT_ID`, `NOKV_BUCKET`, `NOKV_OBJECT_ENDPOINT`, `NOKV_OBJECT_ROOT`, `NOKV_OBJECT_KEY`, `NOKV_OBJECT_SECRET`; the `nokv` SDK importable | `nokv_live_env_missing`, `nokv_coordination_live_not_enabled`, `nokv_sdk_missing` | -| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.0), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` | +| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.1), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` | POSIX-only rows report `unverified/posix_only` on Windows. diff --git a/loopx/control_plane/coordination/nokv_authority_store.ts b/loopx/control_plane/coordination/nokv_authority_store.ts index f25084167a..a9feaaf915 100644 --- a/loopx/control_plane/coordination/nokv_authority_store.ts +++ b/loopx/control_plane/coordination/nokv_authority_store.ts @@ -54,6 +54,12 @@ export interface NoKVBlobCasRequest { bytes: Uint8Array; operation_id: string; artifact_revision_id: string; + /** + * The workbench incarnation this publication is bound to. NoKV evaluates it + * atomically with `expected_generation` before any durable row or object + * exists; a stale value is refused as `failed/store_identity_mismatch`. + */ + expected_workspace_incarnation_id: string; } export type NoKVBlobCasResult = @@ -186,6 +192,11 @@ function readFailure(error: unknown): AuthorityStoreReadFailure { }; } +/** The incarnation half of a validated `nokv:{workbench}:{incarnation}` identity. */ +function boundIncarnation(storeIdentity: string, workbench: string): string { + return storeIdentity.slice(`nokv:${workbench}:`.length); +} + function validStoreIdentity(value: string, workbench: string): boolean { const prefix = `nokv:${workbench}:`; return value.startsWith(prefix) && HEX_128_PATTERN.test(value.slice(prefix.length)); @@ -402,6 +413,10 @@ export class NoKVAuthorityStore implements AuthorityStore { // attempt. Keep the LoopX operation id stable in the authority envelope, // while giving each physical retry a fresh pair of lower-layer ids. A // response-lost success is still settled only by reading that envelope. + // The request also names the incarnation the envelope was read from, so a + // workbench restored to a new incarnation between this read and the + // publish refuses the write instead of accepting it at a restarted + // generation. const attemptNonce = randomUUID(); let result: NoKVBlobCasResult; try { @@ -409,6 +424,7 @@ export class NoKVAuthorityStore implements AuthorityStore { workbench: this.workbench, path: this.path, expected_generation: expectedGeneration, + expected_workspace_incarnation_id: boundIncarnation(current.identity, this.workbench), bytes: payload, operation_id: physicalAttemptIdentity( "operation", @@ -433,11 +449,11 @@ export class NoKVAuthorityStore implements AuthorityStore { }; } if (result.status === "applied" && result.generation === generation) { - // Generation is not a workbench-incarnation fence: NoKV may restart it - // after remove/recreate. Never expose success until a fresh read proves - // this exact transaction in the current incarnation. Preventing the - // stale-incarnation write itself still requires an atomic provider - // primitive that accepts the expected incarnation. + // The incarnation fence removes the stale write, not the readback + // obligation: success is exposed only after a fresh read proves this + // exact transaction in the current incarnation (RFC ยง6.2), so an owner + // that ignored the fence still cannot make a restarted generation look + // like a LoopX commit. return await this.settleCommitFromReadback( normalized.expected_provider_revision, transaction, diff --git a/loopx/control_plane/coordination/nokv_jsonl_helper.py b/loopx/control_plane/coordination/nokv_jsonl_helper.py index 8e11ecd406..97269ce02f 100644 --- a/loopx/control_plane/coordination/nokv_jsonl_helper.py +++ b/loopx/control_plane/coordination/nokv_jsonl_helper.py @@ -4,13 +4,16 @@ receipts, cursor ordering, and ambiguous-outcome reconciliation stay in the TypeScript ``NoKVAuthorityStore``. The first JSON line configures one SDK client; every later line invokes exactly one of ``store_identity``, -``read_blob``, or ``cas_publish_blob``. +``read_blob``, or ``cas_publish_blob``. Every publication names the workbench +incarnation it expects; the SDK refuses a stale one before any row or object +exists, and the helper admits only an SDK that can make that refusal typed. """ from __future__ import annotations import base64 import binascii +import inspect import json import re import sys @@ -19,8 +22,13 @@ _HEX_128 = re.compile(r"^[0-9a-f]{32}$") _CLIENT_AVAILABILITY_ERRORS = (RuntimeError, OSError) -QUALIFIED_NOKV_SDK_VERSION = "0.11.0" +QUALIFIED_NOKV_SDK_VERSION = "0.11.1" QUALIFIED_NOKV_API_VERSION = 1 +# NoKV 0.11.1 evaluates ``expected_workspace_incarnation_id`` atomically with +# the generation before any durable row or object exists and refuses a stale +# incarnation with a typed exception. Both halves are admission requirements. +_INCARNATION_FENCE_PARAMETER = "expected_workspace_incarnation_id" +_INCARNATION_MISMATCH_EXCEPTION = "WorkspaceIncarnationMismatch" _CONFIG_KEYS = frozenset( { @@ -75,10 +83,13 @@ class RequestError(ValueError): class SdkCapabilityMismatch(RequestError): - """The installed NoKV SDK lacks the constructor this configuration needs. + """The installed NoKV SDK lacks a surface this helper requires. - Raised only after the configuration itself validated, so the caller can - tell "wrong wheel for this routing kind" apart from "invalid config". + Either the routing constructor this configuration names or the fenced + publication surface (``expected_workspace_incarnation_id`` plus the typed + ``WorkspaceIncarnationMismatch`` refusal). Raised only after the + configuration itself validated, so the caller can tell "wrong wheel" apart + from "invalid config". """ @@ -313,10 +324,13 @@ def _cas_publish_blob( payload = _decode_bytes(values.get("bytes_base64")) operation_id = _required_string(values, "operation_id") artifact_revision_id = _required_string(values, "artifact_revision_id") + expected_incarnation = _required_string(values, _INCARNATION_FENCE_PARAMETER) if not _HEX_128.fullmatch(operation_id): raise RequestError("operation_id must be 32 lowercase hex") if not _HEX_128.fullmatch(artifact_revision_id): raise RequestError("artifact_revision_id must be 32 lowercase hex") + if not _HEX_128.fullmatch(expected_incarnation): + raise RequestError(f"{_INCARNATION_FENCE_PARAMETER} must be 32 lowercase hex") try: raw_result = client.publish_bytes( workbench, @@ -326,6 +340,24 @@ def _cas_publish_blob( expected_generation=expected_generation, operation_id=operation_id, artifact_revision_id=artifact_revision_id, + expected_workspace_incarnation_id=expected_incarnation, + ) + except _incarnation_mismatch_errors() as error: + # The owner evaluated the fence before claiming the path, so nothing + # durable exists for this attempt. A refusal that names a different + # fence than the one sent is an SDK contract violation, not evidence. + if getattr(error, "expected", None) != expected_incarnation: + return _opaque_failure( + request_id, + "ambiguous", + "provider_protocol_violation", + "NoKV refused a workbench incarnation fence this request did not send", + ) + return _opaque_failure( + request_id, + "failed", + "store_identity_mismatch", + "NoKV workbench incarnation does not match the expected incarnation", ) except FileExistsError: return _response( @@ -533,10 +565,17 @@ def build_client(config_value: object) -> Any: RoutingConfig = nokv.RoutingConfig except AttributeError as error: raise RequestError("the NoKV Python SDK surface is incomplete") from error + if not publish_incarnation_fence_supported(Client): + raise SdkCapabilityMismatch( + "the installed NoKV Python SDK does not fence publication on the " + f"expected workbench incarnation (Client.publish_bytes lacks " + f"{_INCARNATION_FENCE_PARAMETER} or nokv.{_INCARNATION_MISMATCH_EXCEPTION} " + "is missing)" + ) # The kind was checked against _ROUTING_KINDS above, so this attribute - # lookup never reaches an arbitrary caller-chosen name. The 0.11.0 release - # wheel provides etcd/static; the metadata-runtimes line provides seeds. + # lookup never reaches an arbitrary caller-chosen name. The 0.11.x release + # wheels provide etcd/static; the metadata-runtimes line provides seeds. routing_constructor = getattr(RoutingConfig, routing_kind, None) if not callable(routing_constructor): raise SdkCapabilityMismatch( @@ -573,6 +612,35 @@ def build_client(config_value: object) -> Any: raise RequestError("NoKV client configuration is invalid") from error +def _incarnation_mismatch_errors() -> tuple[type[BaseException], ...]: + """The SDK's typed incarnation-fence refusal, or nothing when it has none.""" + error = getattr(sys.modules.get("nokv"), _INCARNATION_MISMATCH_EXCEPTION, None) + if isinstance(error, type) and issubclass(error, BaseException): + return (error,) + return () + + +def publish_incarnation_fence_supported(client_type: object) -> bool: + """True only when publishing names the fence and refusing it is typed. + + Both halves are required: a wheel that accepted the keyword without a typed + refusal would collapse a stale-incarnation rejection into the ambiguous + ``RuntimeError`` path, which is exactly the outcome the fence exists to + remove. The 0.11.0 release wheel has neither half. + """ + if not _incarnation_mismatch_errors(): + return False + try: + signature = inspect.signature(getattr(client_type, "publish_bytes")) + except (AttributeError, TypeError, ValueError): + return False + parameter = signature.parameters.get(_INCARNATION_FENCE_PARAMETER) + return parameter is not None and parameter.kind in ( + inspect.Parameter.POSITIONAL_OR_KEYWORD, + inspect.Parameter.KEYWORD_ONLY, + ) + + def _sdk_protocol_schema() -> str | None: """Return the wire schema the imported SDK declares, if it declares one. diff --git a/loopx/control_plane/coordination/nokv_jsonl_transport.ts b/loopx/control_plane/coordination/nokv_jsonl_transport.ts index 2986b4af31..b8b637f1dd 100644 --- a/loopx/control_plane/coordination/nokv_jsonl_transport.ts +++ b/loopx/control_plane/coordination/nokv_jsonl_transport.ts @@ -308,6 +308,7 @@ export class NoKVJsonLinesTransport implements NoKVBlobTransport { workbench: request.workbench, path: request.path, expected_generation: request.expected_generation, + expected_workspace_incarnation_id: request.expected_workspace_incarnation_id, bytes_base64: Buffer.from(request.bytes).toString("base64"), operation_id: request.operation_id, artifact_revision_id: request.artifact_revision_id, diff --git a/loopx/control_plane/testing/authority_e2e_ladder.py b/loopx/control_plane/testing/authority_e2e_ladder.py index 688683c820..e11e4e3c35 100644 --- a/loopx/control_plane/testing/authority_e2e_ladder.py +++ b/loopx/control_plane/testing/authority_e2e_ladder.py @@ -140,8 +140,14 @@ NOKV_HELPER = Path("loopx") / "control_plane" / "coordination" / "nokv_jsonl_helper.py" NOKV_QUALIFICATION_REPORT_SCHEMA = "loopx_nokv_authority_live_qualification_v0" NOKV_QUALIFICATION_SCOPE = "stage_2a_single_node_store_conformance" -QUALIFIED_NOKV_SDK_VERSION = "0.11.0" +QUALIFIED_NOKV_SDK_VERSION = "0.11.1" QUALIFIED_NOKV_API_VERSION = 1 +# NoKV 0.11.1 refuses a publication fenced on a stale workbench incarnation +# before any durable row or object exists; the live probe must prove it. +NOKV_INCARNATION_FENCE_CHECKS: tuple[str, ...] = ( + "stale_incarnation_fence_rejected", + "stale_incarnation_fence_left_generation_unchanged", +) PROBE_SOURCES: tuple[Path, ...] = ( LIVE_E2E_SCRIPT, TS_READBACK_PROBE, @@ -470,6 +476,10 @@ def _row_nokv_live_qualification(context: RowContext) -> RowOutcome: and all(check.get("status") == "passed" for check in checks if isinstance(check, dict)), "every qualification check must have passed", ) + expect( + all(check_id in check_ids for check_id in NOKV_INCARNATION_FENCE_CHECKS), + "qualification must prove the stale-incarnation publication fence", + ) expect( report.get("nokv_sdk_version") == QUALIFIED_NOKV_SDK_VERSION and report.get("nokv_api_version") == QUALIFIED_NOKV_API_VERSION, @@ -484,6 +494,7 @@ def _row_nokv_live_qualification(context: RowContext) -> RowOutcome: qualification_scope=NOKV_QUALIFICATION_SCOPE, check_count=len(check_ids), check_ids=check_ids, + incarnation_fence_checks=list(NOKV_INCARNATION_FENCE_CHECKS), final_generation=report.get("final_generation"), final_cursor=report.get("final_cursor"), nokv_sdk_version=report.get("nokv_sdk_version"), diff --git a/tests/control_plane/test_shared_goal_authority_e2e.py b/tests/control_plane/test_shared_goal_authority_e2e.py index 1b9564d298..7d92d4f2be 100644 --- a/tests/control_plane/test_shared_goal_authority_e2e.py +++ b/tests/control_plane/test_shared_goal_authority_e2e.py @@ -250,6 +250,18 @@ def test_stage_2a_row_reports_specific_unverified_reasons_for_each_missing_input assert ladder.collect_bindings(inputs)["nokv_client_config_sha256"] is not None +def test_nokv_sdk_pin_and_fence_checks_agree_across_helper_ladder_and_probe() -> None: + from loopx.control_plane.coordination import nokv_jsonl_helper as helper + + assert ladder.QUALIFIED_NOKV_SDK_VERSION == helper.QUALIFIED_NOKV_SDK_VERSION == "0.11.1" + assert ladder.QUALIFIED_NOKV_API_VERSION == helper.QUALIFIED_NOKV_API_VERSION == 1 + probe = (ladder.REPO_ROOT / ladder.NOKV_QUALIFICATION_SCRIPT).read_text(encoding="utf-8") + assert f'export const QUALIFIED_NOKV_SDK_VERSION = "{ladder.QUALIFIED_NOKV_SDK_VERSION}";' in probe + assert f"export const QUALIFIED_NOKV_API_VERSION = {ladder.QUALIFIED_NOKV_API_VERSION};" in probe + for check_id in ladder.NOKV_INCARNATION_FENCE_CHECKS: + assert f'passed("{check_id}")' in probe + + def test_pending_rows_never_exit_green_without_allow_pending( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/control_plane_ts/nokv_authority_store.test.ts b/tests/control_plane_ts/nokv_authority_store.test.ts index 0c08c66c13..a7a98c3057 100644 --- a/tests/control_plane_ts/nokv_authority_store.test.ts +++ b/tests/control_plane_ts/nokv_authority_store.test.ts @@ -31,6 +31,12 @@ interface FakeNoKVBackend { publishFault: PublishFault; casRequests: NoKVBlobCasRequest[]; terminalPhysicalIds: Set; + /** + * Mirrors NoKV 0.11.1: the expected incarnation is evaluated before the path + * is claimed, so a stale fence leaves no row, object, or generation behind. + * `false` models an owner that ignored the fence, to pin the readback guard. + */ + enforceIncarnationFence: boolean; } function fakeBackend(): FakeNoKVBackend { @@ -43,9 +49,14 @@ function fakeBackend(): FakeNoKVBackend { publishFault: null, casRequests: [], terminalPhysicalIds: new Set(), + enforceIncarnationFence: true, }; } +function incarnationOf(identity: string): string { + return identity.slice("nokv:authority-workbench:".length); +} + class FakeNoKVTransport implements NoKVBlobTransport { readonly backend: FakeNoKVBackend; @@ -98,6 +109,16 @@ class FakeNoKVTransport implements NoKVBlobTransport { reason: "physical publication identity is terminal", }; } + if ( + this.backend.enforceIncarnationFence && + request.expected_workspace_incarnation_id !== incarnationOf(this.backend.identity) + ) { + return { + status: "failed", + reason_code: "store_identity_mismatch", + reason: "workbench incarnation does not match the expected incarnation", + }; + } const current = this.backend.blob?.generation ?? null; if (current !== request.expected_generation) { return { status: "conflict", current_generation: current }; @@ -275,8 +296,70 @@ test("NoKV provider fences restored bytes with a different workspace incarnation assert.equal(backend.casRequests.length, callsBeforeRestore); }); -test("NoKV provider does not report applied across a workbench-incarnation race", async () => { +test("NoKV provider names the bound workbench incarnation on every publication", async () => { + const backend = fakeBackend(); + const provider = store(backend); + const created = await provider.commitAuthority(commit(null, "operation-a", 1, 1)); + assert.equal(created.status, "applied"); + const second = commit( + created.status === "applied" ? created.provider_revision : null, + "operation-b", + 2, + 2, + ); + // A terminal physical failure leaves the logical commit ambiguous; the + // caller's retry publishes with fresh lower-layer ids and the same fence. + backend.publishFault = "terminal_ambiguous_before"; + assert.equal((await provider.commitAuthority(second)).status, "ambiguous"); + assert.equal((await provider.commitAuthority(second)).status, "applied"); + + assert.equal(backend.casRequests.length, 3); + assert.notEqual(backend.casRequests[1]?.operation_id, backend.casRequests[2]?.operation_id); + for (const request of backend.casRequests) { + assert.equal(request.expected_workspace_incarnation_id, "a".repeat(32)); + } +}); + +test("NoKV provider is refused before writing across a workbench-incarnation race", async () => { + const backend = fakeBackend(); + const provider = store(backend); + const created = await provider.commitAuthority(commit(null, "operation-a", 1, 1)); + assert.equal(created.status, "applied"); + const bytesBefore = Buffer.from(backend.blob!.bytes).toString("base64"); + const requestsBefore = backend.casRequests.length; + // The store reads generation 1 in incarnation a; the workbench is restored to + // incarnation b before the publish reaches the owner. + backend.rotateIdentityBeforePublish = `nokv:authority-workbench:${"b".repeat(32)}`; + + const result = await provider.commitAuthority( + commit( + created.status === "applied" ? created.provider_revision : null, + "operation-b", + 2, + 2, + ), + ); + + assert.equal(result.status, "failed"); + if (result.status === "failed") { + assert.equal(result.reason_code, "store_identity_mismatch"); + } + assert.equal(backend.casRequests.length, requestsBefore + 1); + assert.equal( + backend.casRequests.at(-1)?.expected_workspace_incarnation_id, + "a".repeat(32), + ); + assert.equal(backend.blob?.generation, 1); + assert.equal(Buffer.from(backend.blob!.bytes).toString("base64"), bytesBefore); + const reloaded = await provider.loadAuthority(); + assert.equal(reloaded.status, "failed"); + if (reloaded.status === "failed") assert.match(reloaded.reason, /lineage mismatch/); + assert.equal((await provider.readReceipt("operation-b")).status, "failed"); +}); + +test("NoKV provider does not report applied when an owner ignores the incarnation fence", async () => { const backend = fakeBackend(); + backend.enforceIncarnationFence = false; const provider = store(backend); backend.rotateIdentityBeforePublish = `nokv:authority-workbench:${"b".repeat(32)}`; @@ -287,6 +370,7 @@ test("NoKV provider does not report applied across a workbench-incarnation race" assert.equal(result.reason_code, "provider_protocol_violation"); assert.match(result.reason, /lineage mismatch/); } + assert.equal(backend.casRequests[0]?.expected_workspace_incarnation_id, "a".repeat(32)); assert.equal((await provider.loadAuthority()).status, "failed"); }); diff --git a/tests/control_plane_ts/nokv_jsonl_transport.test.ts b/tests/control_plane_ts/nokv_jsonl_transport.test.ts index 02a536437d..34c2552df1 100644 --- a/tests/control_plane_ts/nokv_jsonl_transport.test.ts +++ b/tests/control_plane_ts/nokv_jsonl_transport.test.ts @@ -31,7 +31,12 @@ const ETCD_ROUTING = { // NoKV metadata-runtimes line, which drops the etcd constructor. const SEEDS_ROUTING = { kind: "seeds", endpoints: ["127.0.0.1:7750"] }; -async function openSdkHelper(routing: Record = ETCD_ROUTING) { +const FAKE_INCARNATION = "a".repeat(32); + +async function openSdkHelper( + routing: Record = ETCD_ROUTING, + fakeSdkShape?: "0.11.0" | "0.11.1-unfenced", +) { return await NoKVJsonLinesTransport.open({ argv: [PYTHON, SDK_HELPER], config: { @@ -44,6 +49,7 @@ async function openSdkHelper(routing: Record = ETCD_ROUTING) { PYTHONPATH: process.env.PYTHONPATH ? `${FAKE_SDK_ROOT}:${process.env.PYTHONPATH}` : FAKE_SDK_ROOT, + ...(fakeSdkShape === undefined ? {} : { LOOPX_FAKE_NOKV_SDK_SHAPE: fakeSdkShape }), }, request_timeout_ms: 2_000, }); @@ -121,6 +127,7 @@ test("JSON-lines transport starts once and reuses the helper process", async (t) workbench: "authority-workbench", path: "metadata/head.json", expected_generation: null, + expected_workspace_incarnation_id: FAKE_INCARNATION, bytes: Buffer.from("payload", "utf8"), operation_id: "a".repeat(32), artifact_revision_id: "b".repeat(32), @@ -135,6 +142,69 @@ test("JSON-lines transport starts once and reuses the helper process", async (t) } }); +test("JSON-lines helper refuses a stale incarnation fence typed and leaves the generation unchanged", async (t) => { + const transport = await openSdkHelper(); + t.after(async () => await transport.close()); + assert.deepEqual( + await transport.casPublishBlob({ + workbench: "authority-workbench", + path: "metadata/head.json", + expected_generation: null, + expected_workspace_incarnation_id: FAKE_INCARNATION, + bytes: Buffer.from("generation one", "utf8"), + operation_id: "c".repeat(32), + artifact_revision_id: "d".repeat(32), + }), + { status: "applied", generation: 1 }, + ); + + assert.deepEqual( + await transport.casPublishBlob({ + workbench: "authority-workbench", + path: "metadata/head.json", + expected_generation: 1, + expected_workspace_incarnation_id: "b".repeat(32), + bytes: Buffer.from("stale incarnation", "utf8"), + operation_id: "e".repeat(32), + artifact_revision_id: "f".repeat(32), + }), + { + status: "failed", + reason_code: "store_identity_mismatch", + reason: "NoKV workbench incarnation does not match the expected incarnation", + }, + ); + const loaded = await transport.readBlob("authority-workbench", "metadata/head.json"); + assert.equal(loaded.status, "loaded"); + if (loaded.status === "loaded") { + assert.equal(Buffer.from(loaded.bytes).toString("utf8"), "generation one"); + assert.equal(loaded.generation, 1); + } +}); + +test("JSON-lines helper refuses the 0.11.0 wheel by its version pin", async () => { + await assert.rejects( + openSdkHelper(ETCD_ROUTING, "0.11.0"), + (error: unknown) => { + assert.ok(error instanceof NoKVTransportProtocolError); + assert.match(error.message, /must be version 0\.11\.1/); + return true; + }, + ); +}); + +test("JSON-lines helper refuses a 0.11.1-labelled wheel without the publication fence", async () => { + await assert.rejects( + openSdkHelper(ETCD_ROUTING, "0.11.1-unfenced"), + (error: unknown) => { + assert.ok(error instanceof NoKVTransportProtocolError); + assert.match(error.message, /does not fence publication/); + assert.match(error.message, /expected_workspace_incarnation_id/); + return true; + }, + ); +}); + test("JSON-lines helper disconnect is typed unavailable", async (t) => { const transport = await openFaultHelper("disconnect"); t.after(async () => await transport.close()); diff --git a/tests/control_plane_ts/nokv_stage2a_qualification_harness.test.ts b/tests/control_plane_ts/nokv_stage2a_qualification_harness.test.ts index 258e58ee41..1ceef39e8b 100644 --- a/tests/control_plane_ts/nokv_stage2a_qualification_harness.test.ts +++ b/tests/control_plane_ts/nokv_stage2a_qualification_harness.test.ts @@ -36,7 +36,7 @@ const PYTHON = process.env.LOOPX_TEST_PYTHON ?? "python3"; /** Minimal module that satisfies helper admission and records that it was imported. */ const STAND_IN_SDK_SOURCE = `import os -__version__ = "0.11.0" +__version__ = "0.11.1" API_VERSION = 1 _marker = os.environ.get("LOOPX_TEST_STAND_IN_MARKER") @@ -69,9 +69,18 @@ class ObjectStoreConfig: return ("s3", values) +class WorkspaceIncarnationMismatch(RuntimeError): + def __init__(self, message, expected): + super().__init__(message) + self.expected = expected + + class Client: def __init__(self, **values): self.values = values + + def publish_bytes(self, workbench, path, data, *, expected_workspace_incarnation_id=None, **values): + raise NotImplementedError `; function absolutePythonExecutable(): string | null { @@ -86,10 +95,14 @@ function absolutePythonExecutable(): string | null { interface Backend { blob: { bytes: Uint8Array; generation: number } | null; ignoreCas: boolean; + /** Models an owner that accepts a stale incarnation fence instead of refusing it. */ + ignoreIncarnationFence?: boolean; publishCalls: number; pretendAppliedWithoutWriteOnCall: number | null; } +const FAKE_INCARNATION = "a".repeat(32); + class FakeQualificationTransport implements QualificationTransport { readonly backend: Backend; closed = false; @@ -101,7 +114,7 @@ class FakeQualificationTransport implements QualificationTransport { async storeIdentity(workbench: string): Promise { return { status: "available", - store_identity: `nokv:${workbench}:${"a".repeat(32)}`, + store_identity: `nokv:${workbench}:${FAKE_INCARNATION}`, }; } @@ -117,6 +130,16 @@ class FakeQualificationTransport implements QualificationTransport { async casPublishBlob(request: NoKVBlobCasRequest): Promise { this.backend.publishCalls += 1; + if ( + !this.backend.ignoreIncarnationFence && + request.expected_workspace_incarnation_id !== FAKE_INCARNATION + ) { + return { + status: "failed", + reason_code: "store_identity_mismatch", + reason: "workbench incarnation does not match the expected incarnation", + }; + } const current = this.backend.blob?.generation ?? null; if (!this.backend.ignoreCas && current !== request.expected_generation) { return { status: "conflict", current_generation: current }; @@ -191,7 +214,7 @@ test("Stage 2A qualification harness rejects a relative Python executable", () = test("Stage 2A qualification harness names the exact NoKV SDK contract", () => { assert.equal(QUALIFICATION_SCOPE, "stage_2a_single_node_store_conformance"); - assert.equal(QUALIFIED_NOKV_SDK_VERSION, "0.11.0"); + assert.equal(QUALIFIED_NOKV_SDK_VERSION, "0.11.1"); assert.equal(QUALIFIED_NOKV_API_VERSION, 1); }); @@ -216,6 +239,8 @@ test("qualification proves create, ambiguous reconciliation, contention, and fre "fresh_authority_target", "create_applied", "create_generation_one", + "stale_incarnation_fence_rejected", + "stale_incarnation_fence_left_generation_unchanged", "response_lost_success_reconciled", "generation_cas_applied", "generation_two_readback", @@ -232,6 +257,28 @@ test("qualification proves create, ambiguous reconciliation, contention, and fre assert.ok(opened.every((transport) => transport.closed)); }); +test("qualification rejects a backend that accepts a stale incarnation fence", async () => { + const backend: Backend = { + blob: null, + ignoreCas: false, + ignoreIncarnationFence: true, + publishCalls: 0, + pretendAppliedWithoutWriteOnCall: null, + }; + await assert.rejects( + exerciseQualificationSequence(BASE_OPTIONS, async () => + new FakeQualificationTransport(backend)), + (error: unknown) => { + assert.ok(error instanceof QualificationFailure); + assert.equal(error.reasonCode, "stale_incarnation_fence_not_enforced"); + return true; + }, + ); + // The unfenced owner applied the stale write, so the envelope moved to + // generation 2 without a LoopX commit: exactly the outcome the fence removes. + assert.equal(backend.blob?.generation, 2); +}); + test("qualification rejects a backend that does not enforce generation CAS", async () => { const backend: Backend = { blob: null, @@ -285,7 +332,8 @@ test("qualification requires durable readback after the injected response loss", blob: null, ignoreCas: false, publishCalls: 0, - pretendAppliedWithoutWriteOnCall: 2, + // create (1), stale fence probe (2), advance (3) + pretendAppliedWithoutWriteOnCall: 3, }; await assert.rejects( exerciseQualificationSequence(BASE_OPTIONS, async () => diff --git a/tests/fixtures/nokv_fake_sdk/nokv/__init__.py b/tests/fixtures/nokv_fake_sdk/nokv/__init__.py index 52ce1920a0..6844159d6f 100644 --- a/tests/fixtures/nokv_fake_sdk/nokv/__init__.py +++ b/tests/fixtures/nokv_fake_sdk/nokv/__init__.py @@ -1,14 +1,25 @@ from __future__ import annotations +import os from typing import Any -__version__ = "0.11.0" +# The fixture presents the 0.11.1 release surface by default. Two other shapes +# exist so tests can prove the helper refuses them at admission instead of +# downgrading silently: "0.11.0" (the previous release: no publication fence, +# no typed refusal) and "0.11.1-unfenced" (a wheel labelled 0.11.1 that lacks +# the fence surface). +_SHAPE = os.environ.get("LOOPX_FAKE_NOKV_SDK_SHAPE", "0.11.1") +if _SHAPE not in {"0.11.1", "0.11.0", "0.11.1-unfenced"}: + raise RuntimeError(f"unknown fake NoKV SDK shape {_SHAPE!r}") +_FENCED = _SHAPE == "0.11.1" +__version__ = "0.11.0" if _SHAPE == "0.11.0" else "0.11.1" API_VERSION = 1 +_CURRENT_INCARNATION = "a" * 32 class RoutingConfig: - # Union of the two real wheels the helper is qualified against: the 0.11.0 - # release provides etcd/static, the metadata-runtimes line provides seeds. + # Union of the two real wheels the helper is qualified against: the 0.11.x + # releases provide etcd/static, the metadata-runtimes line provides seeds. @staticmethod def seeds(endpoints: list[str]) -> object: return ("seeds", endpoints) @@ -32,6 +43,16 @@ def s3(**values: Any) -> object: return ("s3", values) +if _FENCED: + + class WorkspaceIncarnationMismatch(RuntimeError): + """Mirror of the 0.11.1 SDK refusal: the fence did not match, nothing was written.""" + + def __init__(self, message: str, expected: str) -> None: + super().__init__(message) + self.expected = expected + + class Client: def __init__(self, **_values: Any) -> None: self._bytes: bytes | None = None @@ -43,7 +64,7 @@ def find_workspaces(self, **_values: Any) -> dict[str, Any]: { "workspace": { "workbench": "authority-workbench", - "workspace_incarnation_id": "a" * 32, + "workspace_incarnation_id": _CURRENT_INCARNATION, } } ], @@ -58,7 +79,7 @@ def read(self, workbench: str, path: str) -> dict[str, Any]: "metadata": { "workbench": workbench, "path": path, - "workspace_incarnation_id": "a" * 32, + "workspace_incarnation_id": _CURRENT_INCARNATION, "generation": self._generation, }, } @@ -68,7 +89,23 @@ def publish_bytes( workbench: str, path: str, payload: bytes, + *, + expected_workspace_incarnation_id: str | None = None, **values: Any, + ) -> dict[str, Any]: + # The real owner evaluates the fence before claiming the path, so a + # stale fence leaves the stored generation untouched. + if ( + expected_workspace_incarnation_id is not None + and expected_workspace_incarnation_id != _CURRENT_INCARNATION + ): + raise WorkspaceIncarnationMismatch( + "workbench incarnation mismatch", expected_workspace_incarnation_id + ) + return self._publish(workbench, path, payload, values) + + def _publish( + self, workbench: str, path: str, payload: bytes, values: dict[str, Any] ) -> dict[str, Any]: expected = values["expected_generation"] if expected is None and self._generation is not None: @@ -84,3 +121,13 @@ def publish_bytes( "path": path, "generation": self._generation, } + + +if not _FENCED: + + def _publish_bytes_without_fence( + self: Client, workbench: str, path: str, payload: bytes, **values: Any + ) -> dict[str, Any]: + return self._publish(workbench, path, payload, values) + + Client.publish_bytes = _publish_bytes_without_fence # type: ignore[method-assign] diff --git a/tests/test_nokv_jsonl_helper.py b/tests/test_nokv_jsonl_helper.py index a17f8551de..0c06550b22 100644 --- a/tests/test_nokv_jsonl_helper.py +++ b/tests/test_nokv_jsonl_helper.py @@ -16,10 +16,43 @@ build_client, handle_request, main, + publish_incarnation_fence_supported, serve, ) +class WorkspaceIncarnationMismatch(RuntimeError): + """Test double for ``nokv.WorkspaceIncarnationMismatch`` (0.11.1).""" + + def __init__(self, message: str, expected: str) -> None: + super().__init__(message) + self.expected = expected + + +def qualified_client_class( + constructor: Any = None, +) -> type: + """A ``nokv.Client`` stand-in whose publish surface satisfies admission.""" + + class QualifiedClient: + def __init__(self, **kwargs: Any) -> None: + if constructor is not None: + constructor(**kwargs) + + def publish_bytes( + self, + workbench: str, + path: str, + data: bytes, + *, + expected_workspace_incarnation_id: str | None = None, + **values: Any, + ) -> dict[str, Any]: + raise NotImplementedError + + return QualifiedClient + + class FakeClient: def __init__(self) -> None: self.find_pages: list[dict[str, Any]] = [] @@ -36,8 +69,18 @@ def read(self, *args: Any) -> dict[str, Any]: raise self.read_result return self.read_result - def publish_bytes(self, *args: Any, **kwargs: Any) -> dict[str, Any]: - self.publish_calls.append((args, kwargs)) + def publish_bytes( + self, + *args: Any, + expected_workspace_incarnation_id: str | None = None, + **kwargs: Any, + ) -> dict[str, Any]: + self.publish_calls.append( + ( + args, + {**kwargs, "expected_workspace_incarnation_id": expected_workspace_incarnation_id}, + ) + ) if isinstance(self.publish_result, BaseException): raise self.publish_result return self.publish_result @@ -47,6 +90,22 @@ def request(operation: str, **values: Any) -> dict[str, Any]: return {"request_id": "request-a", "operation": operation, **values} +def publish_request(**overrides: Any) -> dict[str, Any]: + """A well-formed ``cas_publish_blob`` request; every publish names its fence.""" + + values: dict[str, Any] = { + "workbench": "authority-workbench", + "path": "metadata/head.json", + "expected_generation": 4, + "bytes_base64": base64.b64encode(b"{}").decode("ascii"), + "operation_id": "a" * 32, + "artifact_revision_id": "b" * 32, + "expected_workspace_incarnation_id": "c" * 32, + } + values.update(overrides) + return request("cas_publish_blob", **values) + + def identity_page() -> dict[str, Any]: return { "workspaces": [ @@ -186,15 +245,7 @@ def test_cas_publish_blob_forwards_exact_generation_bytes_and_identities() -> No payload = b'{"head":true}' result = handle_request( client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", - expected_generation=4, - bytes_base64=base64.b64encode(payload).decode("ascii"), - operation_id="a" * 32, - artifact_revision_id="b" * 32, - ), + publish_request(bytes_base64=base64.b64encode(payload).decode("ascii")), ) assert result == { @@ -209,6 +260,7 @@ def test_cas_publish_blob_forwards_exact_generation_bytes_and_identities() -> No "expected_generation": 4, "operation_id": "a" * 32, "artifact_revision_id": "b" * 32, + "expected_workspace_incarnation_id": "c" * 32, } @@ -258,18 +310,7 @@ def test_publish_never_reports_applied_for_an_sdk_result_bound_to_another_write( client.publish_result = publish_result(generation=5) client.publish_result[field] = wrong_value - result = handle_request( - client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", - expected_generation=4, - bytes_base64=base64.b64encode(b"{}").decode("ascii"), - operation_id="a" * 32, - artifact_revision_id="b" * 32, - ), - ) + result = handle_request(client, publish_request()) assert result["status"] == "ambiguous" assert result["reason_code"] == "provider_protocol_violation" @@ -278,68 +319,114 @@ def test_publish_never_reports_applied_for_an_sdk_result_bound_to_another_write( def test_cas_publish_blob_maps_only_proven_collision_to_conflict() -> None: client = FakeClient() client.publish_result = FileExistsError("already exists") - conflict = handle_request( - client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", - expected_generation=None, - bytes_base64=base64.b64encode(b"{}").decode("ascii"), - operation_id="a" * 32, - artifact_revision_id="b" * 32, - ), - ) + conflict = handle_request(client, publish_request(expected_generation=None)) assert conflict["status"] == "conflict" assert conflict["current_generation"] is None client.publish_result = RuntimeError("generation conflict or lost response") - ambiguous = handle_request( - client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", - expected_generation=1, - bytes_base64=base64.b64encode(b"{}").decode("ascii"), - operation_id="a" * 32, - artifact_revision_id="b" * 32, - ), - ) + ambiguous = handle_request(client, publish_request(expected_generation=1)) assert ambiguous["status"] == "ambiguous" assert ambiguous["reason_code"] == "nokv_publish_outcome_unknown" assert ambiguous["reason"] == "NoKV publish outcome is unknown" assert "lost response" not in ambiguous["reason"] client.publish_result = ValueError("post-call conversion exposed an endpoint") - malformed = handle_request( - client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", - expected_generation=1, - bytes_base64=base64.b64encode(b"{}").decode("ascii"), - operation_id="a" * 32, - artifact_revision_id="b" * 32, - ), - ) + malformed = handle_request(client, publish_request(expected_generation=1)) assert malformed["status"] == "ambiguous" assert "endpoint" not in malformed["reason"] +def test_stale_incarnation_fence_refusal_is_typed_and_not_ambiguous( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setitem( + sys.modules, + "nokv", + types.SimpleNamespace(WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch), + ) + client = FakeClient() + client.publish_result = WorkspaceIncarnationMismatch( + "workbench 'authority-workbench' is incarnation dddd... at endpoint 127.0.0.1", + "c" * 32, + ) + + refused = handle_request(client, publish_request()) + + assert refused == { + "request_id": "request-a", + "status": "failed", + "reason_code": "store_identity_mismatch", + "reason": "NoKV workbench incarnation does not match the expected incarnation", + } + # The refusal is a RuntimeError subclass; without the typed branch it would + # have been reported as nokv_publish_outcome_unknown. + assert isinstance(client.publish_result, RuntimeError) + + +def test_incarnation_refusal_naming_another_fence_is_a_protocol_violation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setitem( + sys.modules, + "nokv", + types.SimpleNamespace(WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch), + ) + client = FakeClient() + client.publish_result = WorkspaceIncarnationMismatch("mismatch", "d" * 32) + + result = handle_request(client, publish_request()) + + assert result["status"] == "ambiguous" + assert result["reason_code"] == "provider_protocol_violation" + assert "d" * 32 not in result["reason"] + + +def test_incarnation_refusal_without_a_typed_sdk_class_stays_ambiguous( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # Admission never lets a fence-less wheel reach a publish, but the request + # path must not rely on that: with no typed class every RuntimeError is + # still an unknown outcome. + monkeypatch.setitem(sys.modules, "nokv", types.SimpleNamespace()) + client = FakeClient() + client.publish_result = WorkspaceIncarnationMismatch("mismatch", "c" * 32) + + result = handle_request(client, publish_request()) + + assert result["status"] == "ambiguous" + assert result["reason_code"] == "nokv_publish_outcome_unknown" + + +@pytest.mark.parametrize( + "fence", + [None, "", "C" * 32, "c" * 31, "c" * 33, 12, " " + "c" * 31], +) +def test_publish_without_a_valid_incarnation_fence_fails_before_the_sdk_call( + fence: object, +) -> None: + client = FakeClient() + values: dict[str, Any] = {} + if fence is not None: + values["expected_workspace_incarnation_id"] = fence + else: + values["expected_workspace_incarnation_id"] = None + + invalid = handle_request(client, publish_request(**values)) + + assert invalid["status"] == "failed" + assert invalid["reason_code"] == "invalid_request" + assert "expected_workspace_incarnation_id" in invalid["reason"] + assert client.publish_calls == [] + + def test_invalid_publish_request_fails_before_calling_the_sdk() -> None: client = FakeClient() invalid = handle_request( client, - request( - "cas_publish_blob", - workbench="authority-workbench", - path="metadata/head.json", + publish_request( expected_generation=True, bytes_base64="not base64", operation_id="short", - artifact_revision_id="b" * 32, ), ) assert invalid["status"] == "failed" @@ -383,9 +470,10 @@ def static(*args: Any) -> object: return object() module = types.SimpleNamespace( - __version__="0.11.0", + __version__="0.11.1", API_VERSION=1, - Client=lambda **_kwargs: object(), + Client=qualified_client_class(), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=types.SimpleNamespace(memory=lambda: object()), RoutingConfig=RoutingConfig, ) @@ -426,9 +514,10 @@ def unavailable_client(**_kwargs: Any) -> object: raise ValueError("provider endpoint and credential detail") module = types.SimpleNamespace( - __version__="0.11.0", + __version__="0.11.1", API_VERSION=1, - Client=unavailable_client, + Client=qualified_client_class(unavailable_client), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=types.SimpleNamespace(memory=lambda: object()), RoutingConfig=RoutingConfig, ) @@ -474,9 +563,10 @@ def client(**_kwargs: Any) -> object: return object() module = types.SimpleNamespace( - __version__="0.11.0", + __version__="0.11.1", API_VERSION=1, - Client=client, + Client=qualified_client_class(client), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=ObjectStoreConfig, RoutingConfig=RoutingConfig, ) @@ -508,7 +598,7 @@ def client(**_kwargs: Any) -> object: @pytest.mark.parametrize( ("sdk_version", "api_version"), - [("incompatible-version", 1), ("0.11.0", 999)], + [("incompatible-version", 1), ("0.11.1", 999), ("0.11.0", 1)], ) def test_sdk_version_or_api_mismatch_fails_before_provider_construction( monkeypatch: pytest.MonkeyPatch, @@ -519,7 +609,10 @@ def test_sdk_version_or_api_mismatch_fails_before_provider_construction( module = types.SimpleNamespace( __version__=sdk_version, API_VERSION=api_version, - Client=lambda **_kwargs: construction_calls.append("client"), + Client=qualified_client_class( + lambda **_kwargs: construction_calls.append("client") + ), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=types.SimpleNamespace( memory=lambda: construction_calls.append("object_store") ), @@ -548,13 +641,96 @@ def test_sdk_version_or_api_mismatch_fails_before_provider_construction( assert "999" not in str(raised.value) +def _publish_without_fence_parameter( + self: Any, workbench: str, path: str, data: bytes, **values: Any +) -> dict[str, Any]: + raise NotImplementedError + + +@pytest.mark.parametrize( + ("publish_bytes", "typed_refusal"), + [ + # The parameter is absent: accepting the keyword through **values is not + # a capability, the SDK would ignore or mis-handle the fence. + (_publish_without_fence_parameter, WorkspaceIncarnationMismatch), + # The parameter exists but the refusal is untyped: a stale incarnation + # would collapse into the ambiguous RuntimeError path. + (qualified_client_class().publish_bytes, None), + (qualified_client_class().publish_bytes, "not-an-exception-class"), + # Neither half (the 0.11.0 surface under a 0.11.1 label). + (_publish_without_fence_parameter, None), + ], +) +def test_sdk_without_a_typed_publication_fence_is_refused_at_admission( + monkeypatch: pytest.MonkeyPatch, + publish_bytes: Any, + typed_refusal: object, +) -> None: + construction_calls: list[str] = [] + + class Client: + def __init__(self, **_kwargs: Any) -> None: + construction_calls.append("client") + + Client.publish_bytes = publish_bytes # type: ignore[attr-defined] + module = types.SimpleNamespace( + __version__="0.11.1", + API_VERSION=1, + Client=Client, + ObjectStoreConfig=types.SimpleNamespace( + memory=lambda: construction_calls.append("object_store") + ), + RoutingConfig=types.SimpleNamespace( + etcd=lambda *_args: construction_calls.append("routing") + ), + ) + if typed_refusal is not None: + module.WorkspaceIncarnationMismatch = typed_refusal + monkeypatch.setitem(sys.modules, "nokv", module) + + with pytest.raises(SdkCapabilityMismatch) as raised: + build_client( + { + "root_id": "a" * 32, + "routing": { + "kind": "etcd", + "endpoints": ["http://unused.invalid"], + "key_prefix": "/nokv/control", + "lease_ttl_seconds": 10, + }, + "object_store": {"kind": "memory"}, + } + ) + + assert "expected_workspace_incarnation_id" in str(raised.value) + assert "unused.invalid" not in str(raised.value) + assert construction_calls == [] + + +def test_fence_capability_requires_both_the_parameter_and_the_typed_refusal( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.delitem(sys.modules, "nokv", raising=False) + assert publish_incarnation_fence_supported(qualified_client_class()) is False + + monkeypatch.setitem( + sys.modules, + "nokv", + types.SimpleNamespace(WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch), + ) + assert publish_incarnation_fence_supported(qualified_client_class()) is True + assert publish_incarnation_fence_supported(object()) is False + assert publish_incarnation_fence_supported(lambda **_kwargs: object()) is False + + def test_open_handshake_reports_the_qualified_sdk_contract( monkeypatch: pytest.MonkeyPatch, ) -> None: module = types.SimpleNamespace( - __version__="0.11.0", + __version__="0.11.1", API_VERSION=1, - Client=lambda **_kwargs: object(), + Client=qualified_client_class(), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=types.SimpleNamespace(memory=lambda: object()), RoutingConfig=types.SimpleNamespace(etcd=lambda *_args: object()), ) @@ -588,15 +764,16 @@ def test_open_handshake_reports_the_qualified_sdk_contract( "status": "ready", "nokv_api_version": 1, "nokv_protocol_schema": None, - "nokv_sdk_version": "0.11.0", + "nokv_sdk_version": "0.11.1", } def _sdk_module(routing: object, **overrides: Any) -> types.SimpleNamespace: module = types.SimpleNamespace( - __version__="0.11.0", + __version__="0.11.1", API_VERSION=1, - Client=lambda **_kwargs: object(), + Client=qualified_client_class(), + WorkspaceIncarnationMismatch=WorkspaceIncarnationMismatch, ObjectStoreConfig=types.SimpleNamespace(memory=lambda: object()), RoutingConfig=routing, ) @@ -667,7 +844,7 @@ def test_routing_kind_the_sdk_cannot_build_is_a_typed_capability_mismatch( constructed: list[str] = [] module = _sdk_module( sdk_routing, - Client=lambda **_kwargs: constructed.append("client"), + Client=qualified_client_class(lambda **_kwargs: constructed.append("client")), ) module.ObjectStoreConfig = types.SimpleNamespace( memory=lambda: constructed.append("object_store") @@ -762,5 +939,5 @@ def test_open_handshake_echoes_only_a_well_formed_sdk_protocol_schema( "status": "ready", "nokv_api_version": 1, "nokv_protocol_schema": expected, - "nokv_sdk_version": "0.11.0", + "nokv_sdk_version": "0.11.1", }