Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions examples/nokv-authority-store/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,23 @@ This probe intentionally exercises the current Python SDK bridge because it is
the available raw byte-CAS seam. It always starts this checkout's reviewed
`NoKVJsonLinesTransport` and `nokv_jsonl_helper.py`; it has no fake, skip, or
"unverified but successful" CLI path. The helper admits exactly NoKV SDK
`0.11.0` / Python API `1`, and the successful report repeats both values.
`0.11.1` / Python API `1` whose `Client.publish_bytes` names
`expected_workspace_incarnation_id` and whose module exports
`WorkspaceIncarnationMismatch`; the successful report repeats both version
values. A `0.11.0` wheel is refused by the version pin (`invalid_config`), and a
wheel labelled `0.11.1` without that publication surface is refused as
`nokv_sdk_capability_mismatch` before any client is constructed.

## What it proves

Against one **already existing** NoKV workbench, the probe starts three
independent helper processes and verifies:

- the selected tenant/goal path is initially absent and can be created;
- a publication of the generation-1 envelope fenced on a stale workbench
incarnation is refused typed (`store_identity_mismatch`) before any row or
object exists: the stored generation stays 1 and the workbench identity is
unchanged (every LoopX publication carries the incarnation it was read from);
- the stored path generation advances from 1 to 2 under exact generation CAS;
- after the generation-2 CAS lands, an injected response loss is reconciled
from the durable authority envelope and operation receipt rather than from
Expand All @@ -42,30 +51,34 @@ If the SDK, helper, workbench, backend, CAS, or independent readback cannot be
proved, the process exits nonzero. The normal test suite uses deterministic
fakes only to test this sequence and does **not** count as live evidence.

The probe does not prove an atomic expected-incarnation publication fence,
runtime shadow parity, a multi-Agent canary, authority promotion, HA, failover,
restart recovery, capacity, or performance. NoKV generation can restart after
workbench recreation, so the current adapter fails closed through authoritative
post-write readback; preventing the stale-incarnation write itself requires a
future provider primitive. The probe also does not create a workbench. A green
run is Stage 2A single-node storage conformance evidence only.
The probe proves the expected-incarnation publication fence only as far as a
single live owner can show it: a stale fence is refused typed and leaves the
generation untouched. It does not restore or recreate the workbench (NoKV
exposes no client-side retire verb), so the incarnation rotation itself is
covered by NoKV's own executor tests, not by this probe. Success is still
accepted only after authoritative post-write readback in the current
incarnation: the fence removes the stale write, not the readback obligation.
The probe does not prove runtime shadow parity, a multi-Agent canary, authority
promotion, HA, failover, restart recovery, capacity, or performance, and it does
not create a workbench. A green run is Stage 2A single-node storage conformance
evidence only.

## Inputs

Use a current NoKV Python environment. Keep the client configuration in an
ignored local file; do not commit credentials. The helper accepts three routing
kinds and passes each to the matching `RoutingConfig` constructor of the
installed SDK: `etcd` and `static` (the 0.11.0 release wheel) and `seeds`
installed SDK: `etcd` and `static` (the 0.11.x release wheels) and `seeds`
(`{"kind": "seeds", "endpoints": ["IP:PORT", ...]}`, the NoKV
metadata-runtimes line, which names serving owners directly and drops the etcd
constructor). Static routing is valid for a single-node NoKV deployment; etcd
is not required by this probe. A routing kind the installed wheel cannot build
fails the open handshake with `nokv_sdk_capability_mismatch` before any client
is constructed, so a seeds configuration against a 0.11.0 wheel (or an etcd
is constructed, so a seeds configuration against a 0.11.1 wheel (or an etcd
configuration against a metadata-runtimes wheel) is reported as the wrong
wheel, not as an outage. The `ready` handshake echoes `nokv_protocol_schema`:
the SDK's `WORKSPACE_PROTOCOL_SCHEMA` when the wheel exports one, otherwise
`null` (the 0.11.0 release does not). The following shape is illustrative:
`null` (the 0.11.x releases do not). The following shape is illustrative:

```json
{
Expand Down Expand Up @@ -136,7 +149,7 @@ unfenced, pre-existing, or unreadable state exits nonzero with a compact JSON
reason; provider stderr, endpoints, credentials, and raw SDK errors are not
copied into that result. A successful JSON report includes
`"qualification_scope":"stage_2a_single_node_store_conformance"`,
`"nokv_sdk_version":"0.11.0"`, and `"nokv_api_version":1`. The two version
`"nokv_sdk_version":"0.11.1"`, and `"nokv_api_version":1`. The two version
fields are the helper's admission constants: the helper refuses to open a client
for any other SDK version or API version, so a successful report implies them,
but they are not values read back from the NoKV server. The report is Stage
Expand Down
53 changes: 52 additions & 1 deletion examples/nokv-authority-store/live-qualification.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import {

const REPORT_SCHEMA = "loopx_nokv_authority_live_qualification_v0";
export const QUALIFICATION_SCOPE = "stage_2a_single_node_store_conformance";
export const QUALIFIED_NOKV_SDK_VERSION = "0.11.0";
export const QUALIFIED_NOKV_SDK_VERSION = "0.11.1";
export const QUALIFIED_NOKV_API_VERSION = 1;
const REPOSITORY_HELPER = fileURLToPath(
new URL("../../loopx/control_plane/coordination/nokv_jsonl_helper.py", import.meta.url),
Expand Down Expand Up @@ -266,6 +266,16 @@ function applied(
return result;
}

/** A 32-hex incarnation that differs from `current` in its first digit. */
function staleIncarnation(current: string): string {
return `${current.startsWith("0") ? "1" : "0"}${current.slice(1)}`;
}

/** A fresh lower-layer publication identity; the probe never reuses one. */
function freshPhysicalIdentity(): string {
return randomUUID().replaceAll("-", "");
}

async function rawGeneration(
transport: NoKVBlobTransport,
store: NoKVAuthorityStore,
Expand Down Expand Up @@ -361,6 +371,47 @@ export async function exerciseQualificationSequence(
await rawGeneration(firstTransport, first, 1, "create_generation_failed");
passed("create_generation_one");

// A write prepared against one workbench incarnation must not land after
// the workbench is restored to another. Send the generation-1 envelope back
// through the raw transport with a fence naming a different incarnation:
// NoKV must refuse it typed, before any row or object exists, and the
// stored generation and the workbench identity must both be unchanged.
const boundIdentity = firstIdentity.status === "available"
? firstIdentity.store_identity
: fail("workbench_identity_failed", "workbench identity was not available");
const currentEnvelope = await firstTransport.readBlob(first.workbench, first.path);
expect(
currentEnvelope.status === "loaded" && currentEnvelope.generation === 1,
"stale_incarnation_fence_probe_failed",
"the generation-1 envelope could not be read for the fence probe",
);
const staleFence = await firstTransport.casPublishBlob({
workbench: first.workbench,
path: first.path,
expected_generation: 1,
expected_workspace_incarnation_id: staleIncarnation(
boundIdentity.slice(`nokv:${workbench}:`.length),
),
bytes: currentEnvelope.bytes,
operation_id: freshPhysicalIdentity(),
artifact_revision_id: freshPhysicalIdentity(),
});
expect(
staleFence.status === "failed" && staleFence.reason_code === "store_identity_mismatch",
"stale_incarnation_fence_not_enforced",
"NoKV did not refuse a publication fenced on a stale workbench incarnation",
);
passed("stale_incarnation_fence_rejected");
const identityAfterFence = await first.storeIdentity();
expect(
identityAfterFence.status === "available" &&
identityAfterFence.store_identity === boundIdentity,
"stale_incarnation_fence_probe_failed",
"the workbench incarnation changed during the fence probe",
);
await rawGeneration(firstTransport, first, 1, "stale_incarnation_fence_wrote");
passed("stale_incarnation_fence_left_generation_unchanged");

const advanced = applied(
await second.commitAuthority(
commit(created.provider_revision, runId, operationIds.advance, 2, "advance"),
Expand Down
4 changes: 2 additions & 2 deletions examples/shared-goal-authority-e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ suites rather than in the pytest shards.
| `s0.file_matrix_twelve_rows` | 0 | store_direct | deterministic | `examples/nokv-shadow-provider/live_e2e.py` reports exactly the twelve known file-provider scenario rows, all true |
| `s0.nokv_live_matrix` | 0 | store_direct | env:nokv_legacy | the same twelve rows plus `restored_lineage_fails_closed` are true on a live NoKV stack and file/NoKV outcomes are identical |
| `s1.cli_document_decodes_through_ts_store` | 1 | real_cli | deterministic | three CLI writes (`todo add`, `task-lease acquire`, `todo update`) read back through `FileAuthorityStore`: `loadAuthority` loaded at cursor `3`, paged `scanCommitted` yields the three `observation_id`s in order, `readReceipt` finds the first |
| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.0` / API `1`, and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name |
| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.1` / API `1`, the two stale-incarnation fence checks (`stale_incarnation_fence_rejected`, `stale_incarnation_fence_left_generation_unchanged`), and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name |
| `s2b.postgresql_conformance_live` | 2b | store_direct | env:postgresql | `postgresql_authority_store.integration.test.ts` under node's TAP reporter: `# pass >= 9`, `# fail 0`, `# skipped 0` |
| `s2c1.configure_enable_disable_roundtrip` | 2c1 | real_cli | deterministic | `configure-goal` preview does not write, enable writes, captured observations for a todo and a lease, read-back summary `enabled/file_one_way`, disable writes and later writes neither observe nor touch candidate bytes |
| `s2c1.every_writer_family_captures` | 2c1 | real_cli | deterministic | handoff-mode set, todo add/update/complete/supersede/archive-completed, task-lease acquire/renew/transfer each carry `outcome in {captured, replayed, ambiguous_reconciled}`, `primary_writeback_preserved=true`, `provider_to_local_writes=false`, `candidate_read_for_decision=false`; an idempotent re-acquire carries no `authority_shadow`; candidate `cursor == captured count`, operation ids equal observation ids, no time-active lease in the head, head todos equal `todo list` |
Expand Down Expand Up @@ -86,7 +86,7 @@ TypeScript store read.
| `deterministic` | none (needs `node` on `PATH` for the CLI's TypeScript runtime and the read-back probe) | `node_missing` when the probe cannot run |
| `env:postgresql` | `LOOPX_TEST_POSTGRES_URL` plus `node_modules/pg` (`npm ci`) | `postgres_url_missing`, `pg_dependency_missing`, `node_missing` |
| `env:nokv_legacy` | `NOKV_COORDINATION_LIVE=1` and `NOKV_ETCD`, `NOKV_ETCD_PREFIX`, `NOKV_ROOT_ID`, `NOKV_BUCKET`, `NOKV_OBJECT_ENDPOINT`, `NOKV_OBJECT_ROOT`, `NOKV_OBJECT_KEY`, `NOKV_OBJECT_SECRET`; the `nokv` SDK importable | `nokv_live_env_missing`, `nokv_coordination_live_not_enabled`, `nokv_sdk_missing` |
| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.0), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` |
| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.1), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` |

POSIX-only rows report `unverified/posix_only` on Windows.

Expand Down
26 changes: 21 additions & 5 deletions loopx/control_plane/coordination/nokv_authority_store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ export interface NoKVBlobCasRequest {
bytes: Uint8Array;
operation_id: string;
artifact_revision_id: string;
/**
* The workbench incarnation this publication is bound to. NoKV evaluates it
* atomically with `expected_generation` before any durable row or object
* exists; a stale value is refused as `failed/store_identity_mismatch`.
*/
expected_workspace_incarnation_id: string;
}

export type NoKVBlobCasResult =
Expand Down Expand Up @@ -186,6 +192,11 @@ function readFailure(error: unknown): AuthorityStoreReadFailure {
};
}

/** The incarnation half of a validated `nokv:{workbench}:{incarnation}` identity. */
function boundIncarnation(storeIdentity: string, workbench: string): string {
return storeIdentity.slice(`nokv:${workbench}:`.length);
}

function validStoreIdentity(value: string, workbench: string): boolean {
const prefix = `nokv:${workbench}:`;
return value.startsWith(prefix) && HEX_128_PATTERN.test(value.slice(prefix.length));
Expand Down Expand Up @@ -402,13 +413,18 @@ export class NoKVAuthorityStore implements AuthorityStore {
// attempt. Keep the LoopX operation id stable in the authority envelope,
// while giving each physical retry a fresh pair of lower-layer ids. A
// response-lost success is still settled only by reading that envelope.
// The request also names the incarnation the envelope was read from, so a
// workbench restored to a new incarnation between this read and the
// publish refuses the write instead of accepting it at a restarted
// generation.
const attemptNonce = randomUUID();
let result: NoKVBlobCasResult;
try {
result = await this.transport.casPublishBlob({
workbench: this.workbench,
path: this.path,
expected_generation: expectedGeneration,
expected_workspace_incarnation_id: boundIncarnation(current.identity, this.workbench),
bytes: payload,
operation_id: physicalAttemptIdentity(
"operation",
Expand All @@ -433,11 +449,11 @@ export class NoKVAuthorityStore implements AuthorityStore {
};
}
if (result.status === "applied" && result.generation === generation) {
// Generation is not a workbench-incarnation fence: NoKV may restart it
// after remove/recreate. Never expose success until a fresh read proves
// this exact transaction in the current incarnation. Preventing the
// stale-incarnation write itself still requires an atomic provider
// primitive that accepts the expected incarnation.
// The incarnation fence removes the stale write, not the readback
// obligation: success is exposed only after a fresh read proves this
// exact transaction in the current incarnation (RFC §6.2), so an owner
// that ignored the fence still cannot make a restarted generation look
// like a LoopX commit.
return await this.settleCommitFromReadback(
normalized.expected_provider_revision,
transaction,
Expand Down
Loading
Loading