diff --git a/.changeset/runtime-entitlement-withheld.md b/.changeset/runtime-entitlement-withheld.md new file mode 100644 index 00000000..d60d231e --- /dev/null +++ b/.changeset/runtime-entitlement-withheld.md @@ -0,0 +1,5 @@ +--- +"@taskless/cli": patch +--- + +`taskless check` now exits 1 when the Taskless service withholds a runtime rule because the organization's plan does not include runtime rules, instead of printing a notice and exiting 0. The output names the withheld rules and links to the upgrade page, and `--json` carries an `entitlement` object. If a CI job starts failing with this, the rules did not stop matching: they stopped running, and the fix is the plan, not the code. Unauthenticated and `--anonymous` runs, and `sg` and `vale` rules, are unchanged. diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/.openspec.yaml b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/.openspec.yaml new file mode 100644 index 00000000..7f2ad572 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-27 diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/design.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/design.md new file mode 100644 index 00000000..52371e46 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/design.md @@ -0,0 +1,86 @@ +## Context + +`planRuntime` (`rules/runtime/plan.ts`) turns a reconcile outcome into +`{ execute, skipped, notices }`. `selectBlessedRuntimeRules` splits signed rules +into `blessed` (signature in `run`) and `withheld` (everything else), and every +`withheld` rule gets the same skip reason. `check` derives its exit code from +`runEngines` alone, so no skip, of any kind, can fail a run. + +That was correct while every non-`run` outcome was either advisory (`unknown`, +`missing`) or self-repairing (`unsafe`, restored for the next run). An +entitlement withhold is neither: it is the server deliberately declining to run +a rule the user believes is protecting them, and it will keep declining on every +run until someone acts. + +## Decisions + +### 1. Withheld fails the run; the degrade paths still do not + +The standing spec says `check` SHALL NOT change the exit code because runtime +rules were skipped on an **unverified** path (logged out, `--anonymous`, +reconcile unreachable). Withheld is not that. Reconcile completed, the server +answered, and the answer was "these will not run for you". Failing on it does +not reopen the question of whether an outage should break CI. + +Exit code **1**, the code `check` already uses for error findings and engine +failures. A distinct code was considered and rejected: every existing CI recipe +treats non-zero as failure, the JSON envelope's `entitlement` field is the +machine-readable discriminator, and a new code is a contract we would have to +keep. + +`success` in `--json` is `false` whenever the exit code is non-zero, as today. + +### 2. Match withheld entries to local rules by reported path + +`entitlement.withheld` carries `{ ruleId, file }` and no signature, so the +signature join `run` uses is not available. `file` is the path the CLI itself +reported (the reported-path requirement already makes it contractual), so the +join is exact: a signed rule whose reported `check.ts` path is in `withheld` +is withheld for entitlement; every other non-`run` rule keeps today's reason. + +A `withheld` entry that matches no reported file is ignored for the skip list +but still counts toward failing the run. The server said something will not +run; the CLI not being able to name it locally is not a reason to go green. + +### 3. The entitlement object is parsed, not trusted + +`reconcile` normalizes `entitlement` into +`{ runtimeSignatures: false; reason?; upgradeUrl?; withheld: [...] } | undefined`. +It is `undefined` when the field is absent, not an object, or +`runtimeSignatures` is not literally `false`. `withheld` entries without a string +`file` are dropped. `upgradeUrl` is surfaced only when it parses as an absolute +`https:` URL, so a malformed value is never printed as a link. + +Only `withheld` drives the exit code. `runtimeSignatures: false` with an empty +`withheld` (an unentitled org reporting no matching runtime files) passes, since +nothing the user holds was declined. + +### 4. One message, printed once + +Human output prints one notice per run naming the reason, the withheld rule +names, and the upgrade URL, rather than repeating the URL per rule. Each rule's +`skipped` entry gets the short reason `"not included in your Taskless plan"` +so `--json` consumers reading `skipped` alone still see the right cause. + +### 5. Write-time warnings ride the existing notice channels + +`rule create` / `rule improve` already collect `notices` and print them as +`Warning:` in human mode; restore notices already flow into `plan.notices`. The +entitlement warning is one more entry in each, so no new output channel and no +schema change beyond `check`'s `entitlement` field. + +The restore and retrieval responses are typed from the generated schema, which +does not yet carry `entitlement`. They are widened with `MayCarryEntitlement` +(the field as optional `unknown`) and read through the same normalizer as +reconcile, so the CLI ships assuming the field MIGHT be there instead of waiting +on #207's deploy. Tightening that once the service always sends it is #409. + +## Risks + +- **A server bug that lists a file in `withheld` for an entitled org fails CI.** + Accepted: failing closed is the property runtime rules exist for, and the + failure names its cause and the URL to check. +- **The vendored schema is stale in an unrelated way.** A refresh on 2026-09-27 + also changes `successCases`/`failureCases` and drops `signature` from the + `sg`/`vale` file-set variants (the `runtime` variant keeps it). The cloud will + restore those signatures; regeneration waits for that (#409). diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/proposal.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/proposal.md new file mode 100644 index 00000000..bdb6ec24 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/proposal.md @@ -0,0 +1,98 @@ +## Why + +A runtime rule that stops running must never produce a green `check`. That is +the whole reason runtime rules are gated on a server signature, and #403 is the +one path where the CLI breaks it. + +The paid-cloud launch (taskless/taskless#207, `signature-entitlement`) stops +blessing runtime rules for an organization whose plan lacks runtime signatures, +for example after a paid plan lapses. The server keeps reconcile at **HTTP 200** +and adds an additive `entitlement` object, so older CLIs keep working: + +```ts +entitlement: { + runtimeSignatures: boolean; + reason?: "RUNTIME_SIGNATURES_NOT_IN_PLAN"; + upgradeUrl?: string; // absolute, carries from=reconcile + withheld?: { ruleId: string; file: string }[]; +} +``` + +A withheld file appears in `withheld` and in none of `run`, `unsafe`, +`unknown`, or `missing`. Restore and request retrieval carry the same object, +without `withheld`, whenever they return a runtime file set. + +Every CLI through 0.11.2 reads only the four arrays. A withheld rule is absent +from `run`, so it is skipped with the generic reason "not blessed by the server +(unsafe / unknown / drift)", printed as a notice, and `check` exits 0. A +customer whose plan lapses gets a green `taskless check` while their runtime +rules have silently stopped running, and the one line that mentions it blames +tampering that did not happen. + +The restore path has the mirror-image defect. Its completion notice promises +that "the next `check` reports the repaired signature and is blessed through the +ordinary path", which is false for an organization whose plan will not bless it. + +## What Changes + +- **`check` exits non-zero when `entitlement.withheld` is non-empty**, in human + and `--json` modes. Withheld is a verified outcome (reconcile completed and + answered), so it is not one of the degrade paths that deliberately never fail. +- **`check` names the cause.** A withheld rule's skip reason says the plan does + not include runtime rules, not "unsafe / unknown / drift". The human output + prints `reason` and `upgradeUrl` once; `--json` carries an additive + `entitlement` object with `runtimeSignatures`, `reason`, `upgradeUrl`, and the + withheld rule names. +- **The restore-completion notice stops promising blessing** when the restore + response carries `entitlement.runtimeSignatures: false`, and says instead that + the bytes were restored but will not run on this plan. +- **`rule create` and `rule improve` warn when they write a runtime rule** from + a response carrying `entitlement.runtimeSignatures: false`: the rule is on + disk but will not run on this plan. The warning rides the existing `notices` + channel so `--json` callers see it too. +- **The reconcile client parses `entitlement` defensively.** Absent, malformed, + or `runtimeSignatures: true` all mean "no entitlement outcome", so a server + that predates #207 changes nothing. + +Nothing changes for unauthenticated or `--anonymous` runs, for +`--dangerously-run-scripts`, for the degrade paths (reconcile unreachable, +401, no remote), or for `sg` and `vale` rules. + +## Capabilities + +### Modified Capabilities + +- `cli-check`: the exit-code requirement gains the withheld case; a new + requirement defines how a withheld rule is reported. +- `cli-rule-reconciliation`: a new requirement defines how the CLI reads the + `entitlement` object and treats `withheld` as a disposition distinct from the + four buckets. +- `cli-generated-rule-delivery`: a new requirement makes a runtime rule written + under a plan without runtime signatures say so, at write time and on restore. + +## Impact + +- `packages/cli/src/api/reconcile.ts`: parse `entitlement`. +- `packages/cli/src/rules/runtime/plan.ts`, `run-set.ts`: classify withheld + rules, carry the entitlement onto the plan, fix the restore notice. +- `packages/cli/src/commands/check.ts`, `schemas/check.ts`: exit code, human + output, `--json` field. +- `packages/cli/src/commands/rules.ts`: write-time warning for create/improve. +- `packages/cli/src/api/restore.ts`: surface `entitlement` from the restore body. +- `packages/cli/src/agent/check.md`, `agent/ci.md`: document the new exit + condition, which a CI recipe must not read as a findings failure. +- `api.schema.json` / `api.d.ts`: **not** regenerated here. #207 is not + deployed; the live `__schema` carries no `entitlement` today (measured + 2026-09-27). Restore and retrieval type it as a field that might be present + (`MayCarryEntitlement`); tightening once it always is, is #409. + +## Delivery shape + +**Single PR.** The change is one behavior (a withheld rule fails the run and +says why) expressed at four call sites, plus the spec delta and tests, well +inside the ~1200-line guidance. It is independently safe to ship before the +server: with no `entitlement` in the response, every new branch is dead and +behavior is byte-identical to today, which is also the property that lets it +ship ahead of taskless/taskless#210 as the issue asks. + +Fixes #403 diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-check/spec.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-check/spec.md new file mode 100644 index 00000000..32839b9f --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-check/spec.md @@ -0,0 +1,65 @@ +## MODIFIED Requirements + +### Requirement: Check subcommand exit codes reflect error severity + +The CLI SHALL exit with code 0 when no error-severity matches are found (including when only warnings, info, or hints exist) and no runtime rule was withheld for entitlement. The CLI SHALL exit with code 1 when at least one error-severity match is found. The CLI SHALL also exit with code 1 when reconciliation completed and the response's `entitlement.withheld` is non-empty, whatever the findings, in both human and `--json` modes. + +#### Scenario: Exit 0 when clean + +- **WHEN** the scanner produces zero results +- **THEN** the process SHALL exit with code 0 + +#### Scenario: Exit 0 when only warnings + +- **WHEN** the scanner produces results but none have severity "error" +- **THEN** the process SHALL exit with code 0 + +#### Scenario: Exit 1 when errors found + +- **WHEN** the scanner produces at least one result with severity "error" +- **THEN** the process SHALL exit with code 1 + +#### Scenario: Exit 1 when a runtime rule is withheld for entitlement + +- **WHEN** reconciliation returns a non-empty `entitlement.withheld` and the scan produces zero results +- **THEN** the process SHALL exit with code 1 +- **AND** under `--json`, `success` SHALL be `false` + +#### Scenario: Entitlement without a withheld file does not fail + +- **WHEN** reconciliation returns `entitlement.runtimeSignatures: false` with an empty or absent `withheld`, and the scan produces zero results +- **THEN** the process SHALL exit with code 0 + +## ADDED Requirements + +### Requirement: Check reports runtime rules withheld for entitlement as a plan outcome + +When reconciliation completes and returns a non-empty `entitlement.withheld`, `taskless check` SHALL NOT execute any withheld rule, SHALL report each local runtime rule whose reported `check.ts` path appears in `withheld` as skipped with a reason stating that runtime rules are not included in the organization's plan, and SHALL NOT describe it as unsafe, unknown, drifted, or tampered. The human output SHALL include one notice naming the withheld rules, the `entitlement.reason`, and the `entitlement.upgradeUrl` when present. Under `--json`, the output SHALL carry an additive, optional `entitlement` object with `runtimeSignatures`, `reason`, `upgradeUrl`, and `withheld` (the local rule names), present only when reconciliation returned `runtimeSignatures: false`. This is a verified outcome and SHALL NOT be treated as one of the unverified paths that leave the exit code unchanged. + +#### Scenario: Withheld rule is named with its cause + +- **WHEN** an authenticated `check` reconciles and `entitlement.withheld` lists the reported `check.ts` of runtime rule `no-env-leak` +- **THEN** `no-env-leak` SHALL NOT execute +- **AND** its skip reason SHALL state that runtime rules are not included in the plan +- **AND** its skip reason SHALL NOT mention unsafe, unknown, or drift + +#### Scenario: Upgrade URL is shown once + +- **WHEN** two runtime rules are withheld and `entitlement.upgradeUrl` is present +- **THEN** the human output SHALL print the upgrade URL exactly once, in one notice naming both rules and the reason + +#### Scenario: Entitlement appears under --json + +- **WHEN** a runtime rule is withheld and `--json` is set +- **THEN** stdout SHALL include `entitlement` with `runtimeSignatures: false`, `reason`, `upgradeUrl`, and `withheld` naming the rule +- **AND** `skipped` SHALL still list the rule with its plan reason + +#### Scenario: A server without the entitlement object is unchanged + +- **WHEN** reconciliation returns the four arrays and no `entitlement` object +- **THEN** `check` SHALL behave exactly as before this change, including its exit code and skip reasons + +#### Scenario: Degrade paths still never fail + +- **WHEN** `check` runs logged out, with `--anonymous`, or reconciliation cannot complete +- **THEN** the exit code SHALL NOT change because runtime rules were skipped, as before this change diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-generated-rule-delivery/spec.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-generated-rule-delivery/spec.md new file mode 100644 index 00000000..7c8bb470 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-generated-rule-delivery/spec.md @@ -0,0 +1,28 @@ +## ADDED Requirements + +### Requirement: A runtime rule written under a plan without runtime signatures says it will not run + +When the CLI writes a runtime rule from a response whose `entitlement.runtimeSignatures` is exactly `false` (from `rule create`, `rule improve`, or a restore during `check`), it SHALL still write the rule, and SHALL emit a warning that the rule is on disk but will not run on the organization's current plan, including the `upgradeUrl` when it is an absolute `https:` URL. The warning SHALL be carried in the command's existing notices, so it appears in human output and under `--json`. A restore under such a response SHALL NOT state or imply that the next `check` will bless or run the rule. A response with no `entitlement` object, or with `runtimeSignatures: true`, SHALL produce no such warning. + +#### Scenario: Created runtime rule under an unentitled plan + +- **WHEN** `rule create` receives a generated runtime rule with `entitlement.runtimeSignatures: false` +- **THEN** the CLI SHALL write the rule +- **AND** SHALL warn that it will not run on the current plan +- **AND** under `--json` the warning SHALL appear in `notices` + +#### Scenario: Restore under an unentitled plan does not promise blessing + +- **WHEN** a restore during `check` writes a runtime rule and the restore response carries `entitlement.runtimeSignatures: false` +- **THEN** the restore notice SHALL say the rule's bytes were restored but will not run on the current plan +- **AND** SHALL NOT say the next `check` blesses it + +#### Scenario: Entitled or legacy responses do not warn + +- **WHEN** a runtime rule is written from a response with no `entitlement` object or with `runtimeSignatures: true` +- **THEN** the CLI SHALL emit no entitlement warning + +#### Scenario: Static rules never warn + +- **WHEN** `rule create` writes an `sg` or `vale` rule +- **THEN** the CLI SHALL emit no entitlement warning diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-rule-reconciliation/spec.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-rule-reconciliation/spec.md new file mode 100644 index 00000000..87c6b039 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/specs/cli-rule-reconciliation/spec.md @@ -0,0 +1,32 @@ +## ADDED Requirements + +### Requirement: The CLI reads the reconcile entitlement object + +The CLI SHALL read the optional `entitlement` object on a successful reconcile response. It SHALL treat the organization as unentitled only when `entitlement.runtimeSignatures` is exactly `false`; an absent, malformed, or `true` value SHALL be treated as no entitlement outcome, leaving the four buckets to drive execution exactly as before. For an unentitled response the CLI SHALL read `reason`, `upgradeUrl`, and `withheld` (a list of `{ ruleId, file }`), SHALL drop a `withheld` entry without a string `file`, and SHALL surface `upgradeUrl` only when it is an absolute `https:` URL. + +`entitlement.withheld` SHALL be a disposition distinct from the four buckets: a withheld file SHALL NOT be executed, SHALL NOT be surfaced as tamper, drift, or never-issued, and SHALL NOT be sent to restore. The CLI SHALL match a withheld entry to a local runtime rule by the `file` it reported, since the entry carries no signature. A withheld entry that matches no reported file SHALL still count as withheld for the purpose of the exit code. + +#### Scenario: Absent entitlement changes nothing + +- **WHEN** a reconcile response carries no `entitlement` object +- **THEN** the CLI SHALL drive execution from `run`, `unsafe`, `unknown`, and `missing` exactly as before + +#### Scenario: Entitled response changes nothing + +- **WHEN** a reconcile response carries `entitlement: { runtimeSignatures: true }` +- **THEN** the CLI SHALL drive execution from the four buckets exactly as before + +#### Scenario: Withheld is matched by reported path + +- **WHEN** `entitlement.withheld` lists `{ ruleId, file }` and `file` equals the path the CLI reported for a runtime rule's `check.ts` +- **THEN** the CLI SHALL classify that rule as withheld for entitlement and SHALL NOT execute it + +#### Scenario: Withheld is not repaired + +- **WHEN** a runtime rule is withheld for entitlement +- **THEN** the CLI SHALL NOT request a restore for it + +#### Scenario: A malformed upgrade URL is not shown + +- **WHEN** `entitlement.upgradeUrl` is not an absolute `https:` URL +- **THEN** the CLI SHALL omit it from human and `--json` output diff --git a/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/tasks.md b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/tasks.md new file mode 100644 index 00000000..aa323a98 --- /dev/null +++ b/openspec/changes/archive/2026-09-27-runtime-entitlement-withheld/tasks.md @@ -0,0 +1,72 @@ +## 1. Spec + +- [x] 1.1 Dry-run `pnpm openspec archive runtime-entitlement-withheld` against a + WIP commit and confirm every scenario title in `cli-check`, + `cli-rule-reconciliation`, and `cli-generated-rule-delivery` survives, plus + the added ones. The `cli-check` exit-code requirement is MODIFIED under its + unchanged title and restates all three standing scenarios. + +## 2. Parse the entitlement + +- [x] 2.1 Add a `parseEntitlement(value: unknown)` normalizer beside + `reconcile` returning `Entitlement | undefined` per design decision 3 + (`runtimeSignatures === false` only; drop `withheld` entries without a + string `file`; keep `upgradeUrl` only as an absolute `https:` URL). +- [x] 2.2 Carry `entitlement?: Entitlement` on `ReconcileResponse`, populated by + `reconcile`. +- [x] 2.3 Surface `entitlement` on `RestoreOutcome` `ok` from the raw restore + body with the same normalizer; do not hand-widen the generated type. +- [x] 2.4 Unit-test the normalizer: absent, `true`, `false` with and without + `withheld`, malformed entries, non-https and relative URLs. + +## 3. Check fails and explains + +- [x] 3.1 Split `selectBlessedRuntimeRules`' non-`run` rules into + entitlement-withheld (reported path in `entitlement.withheld`) and the + rest; give the former the plan skip reason and keep today's reason for the + latter. +- [x] 3.2 Exclude entitlement-withheld rules from restore targets (the server + already omits them from `missing`; guard anyway so an `unsafe` listing + cannot race a withhold into a repair). +- [x] 3.3 Carry the normalized entitlement on `RuntimePlan`, and add one plan + notice naming the withheld rules, reason, and upgrade URL. +- [x] 3.4 In `check`, force exit code 1 and `success: false` when the plan's + entitlement has a non-empty `withheld`; add the optional `entitlement` + field to `schemas/check.ts` and emit it under `--json`. +- [x] 3.5 Tests: withheld-only run exits 1 with the notice; mixed run executes + blessed rules and still exits 1; no `entitlement` is byte-identical to + today; `runtimeSignatures: false` with empty `withheld` exits 0; degrade + paths unchanged. + +## 4. Write-time warnings + +- [x] 4.1 Restore notice in `repairWithheldRules`: when the restore outcome + carries `runtimeSignatures: false`, replace the "blessed through the + ordinary path" sentence with one saying the bytes were restored but will + not run on the current plan. +- [x] 4.2 `rule create` / `rule improve`: when the generated status carries + `runtimeSignatures: false` and a written rule is a runtime rule, push the + warning into `notices` (read defensively off the body until the schema + carries it). +- [x] 4.3 Tests for both, including static rules and legacy responses emitting + nothing. + +## 5. Docs and release + +- [x] 5.1 Update `agent/check.md` and `agent/ci.md`: a withheld runtime rule + exits 1 with `entitlement` in `--json`, and a CI recipe should report it + as a plan problem rather than a findings failure. +- [x] 5.2 Changeset (`patch`, pre-1.0) stating the exit-code change and what a + CI owner does about it. +- [x] 5.3 `pnpm typecheck` and `pnpm lint`. +- [x] 5.4 Archive the change on this PR. + +## 6. Follow-up (tracked outside this change) + +- [x] 6.1 Ship the entitlement as a field that MIGHT be present + (`MayCarryEntitlement`), so this change does not wait on + taskless/taskless#207. Regenerating the schema and tightening the types + once the service always returns `entitlement` and static file-set + signatures is its own feature: #409. +- [x] 6.2 Gating `--dangerously-run-scripts` on a TTY or `CI=1`, so an agent + cannot add it to get green: #408. diff --git a/openspec/specs/cli-check/spec.md b/openspec/specs/cli-check/spec.md index f305bc5a..8f07df2c 100644 --- a/openspec/specs/cli-check/spec.md +++ b/openspec/specs/cli-check/spec.md @@ -113,7 +113,7 @@ When the `--json` flag is set, the CLI SHALL output each `CheckResult` as a JSON ### Requirement: Check subcommand exit codes reflect error severity -The CLI SHALL exit with code 0 when no error-severity matches are found (including when only warnings, info, or hints exist). The CLI SHALL exit with code 1 when at least one error-severity match is found. +The CLI SHALL exit with code 0 when no error-severity matches are found (including when only warnings, info, or hints exist) and no runtime rule was withheld for entitlement. The CLI SHALL exit with code 1 when at least one error-severity match is found. The CLI SHALL also exit with code 1 when reconciliation completed and the response's `entitlement.withheld` is non-empty, whatever the findings, in both human and `--json` modes. #### Scenario: Exit 0 when clean @@ -130,6 +130,17 @@ The CLI SHALL exit with code 0 when no error-severity matches are found (includi - **WHEN** the scanner produces at least one result with severity "error" - **THEN** the process SHALL exit with code 1 +#### Scenario: Exit 1 when a runtime rule is withheld for entitlement + +- **WHEN** reconciliation returns a non-empty `entitlement.withheld` and the scan produces zero results +- **THEN** the process SHALL exit with code 1 +- **AND** under `--json`, `success` SHALL be `false` + +#### Scenario: Entitlement without a withheld file does not fail + +- **WHEN** reconciliation returns `entitlement.runtimeSignatures: false` with an empty or absent `withheld`, and the scan produces zero results +- **THEN** the process SHALL exit with code 0 + ### Requirement: Check subcommand respects global working directory flag The `check` subcommand SHALL use the resolved working directory from the global `-d` flag (or `process.cwd()` if not specified) as the target directory for `.taskless/` validation and scanner execution. @@ -493,3 +504,35 @@ A notice SHALL NOT affect the exit code. - **WHEN** a `check` run produces no notices - **THEN** human output SHALL print no `Notice: ` line - **AND** `--json` SHALL omit the `notices` field + +### Requirement: Check reports runtime rules withheld for entitlement as a plan outcome + +When reconciliation completes and returns a non-empty `entitlement.withheld`, `taskless check` SHALL NOT execute any withheld rule, SHALL report each local runtime rule whose reported `check.ts` path appears in `withheld` as skipped with a reason stating that runtime rules are not included in the organization's plan, and SHALL NOT describe it as unsafe, unknown, drifted, or tampered. The human output SHALL include one notice naming the withheld rules, the `entitlement.reason`, and the `entitlement.upgradeUrl` when present. Under `--json`, the output SHALL carry an additive, optional `entitlement` object with `runtimeSignatures`, `reason`, `upgradeUrl`, and `withheld` (the local rule names), present only when reconciliation returned `runtimeSignatures: false`. This is a verified outcome and SHALL NOT be treated as one of the unverified paths that leave the exit code unchanged. + +#### Scenario: Withheld rule is named with its cause + +- **WHEN** an authenticated `check` reconciles and `entitlement.withheld` lists the reported `check.ts` of runtime rule `no-env-leak` +- **THEN** `no-env-leak` SHALL NOT execute +- **AND** its skip reason SHALL state that runtime rules are not included in the plan +- **AND** its skip reason SHALL NOT mention unsafe, unknown, or drift + +#### Scenario: Upgrade URL is shown once + +- **WHEN** two runtime rules are withheld and `entitlement.upgradeUrl` is present +- **THEN** the human output SHALL print the upgrade URL exactly once, in one notice naming both rules and the reason + +#### Scenario: Entitlement appears under --json + +- **WHEN** a runtime rule is withheld and `--json` is set +- **THEN** stdout SHALL include `entitlement` with `runtimeSignatures: false`, `reason`, `upgradeUrl`, and `withheld` naming the rule +- **AND** `skipped` SHALL still list the rule with its plan reason + +#### Scenario: A server without the entitlement object is unchanged + +- **WHEN** reconciliation returns the four arrays and no `entitlement` object +- **THEN** `check` SHALL behave exactly as before this change, including its exit code and skip reasons + +#### Scenario: Degrade paths still never fail + +- **WHEN** `check` runs logged out, with `--anonymous`, or reconciliation cannot complete +- **THEN** the exit code SHALL NOT change because runtime rules were skipped, as before this change diff --git a/openspec/specs/cli-generated-rule-delivery/spec.md b/openspec/specs/cli-generated-rule-delivery/spec.md index aefb4b86..438f89db 100644 --- a/openspec/specs/cli-generated-rule-delivery/spec.md +++ b/openspec/specs/cli-generated-rule-delivery/spec.md @@ -46,3 +46,30 @@ whose destination the client computed itself. - **WHEN** a delivered file declares a path containing `..` or an absolute path - **THEN** the CLI SHALL refuse the entire rule - **AND** SHALL NOT have created any file or directory for it + +### Requirement: A runtime rule written under a plan without runtime signatures says it will not run + +When the CLI writes a runtime rule from a response whose `entitlement.runtimeSignatures` is exactly `false` (from `rule create`, `rule improve`, or a restore during `check`), it SHALL still write the rule, and SHALL emit a warning that the rule is on disk but will not run on the organization's current plan, including the `upgradeUrl` when it is an absolute `https:` URL. The warning SHALL be carried in the command's existing notices, so it appears in human output and under `--json`. A restore under such a response SHALL NOT state or imply that the next `check` will bless or run the rule. A response with no `entitlement` object, or with `runtimeSignatures: true`, SHALL produce no such warning. + +#### Scenario: Created runtime rule under an unentitled plan + +- **WHEN** `rule create` receives a generated runtime rule with `entitlement.runtimeSignatures: false` +- **THEN** the CLI SHALL write the rule +- **AND** SHALL warn that it will not run on the current plan +- **AND** under `--json` the warning SHALL appear in `notices` + +#### Scenario: Restore under an unentitled plan does not promise blessing + +- **WHEN** a restore during `check` writes a runtime rule and the restore response carries `entitlement.runtimeSignatures: false` +- **THEN** the restore notice SHALL say the rule's bytes were restored but will not run on the current plan +- **AND** SHALL NOT say the next `check` blesses it + +#### Scenario: Entitled or legacy responses do not warn + +- **WHEN** a runtime rule is written from a response with no `entitlement` object or with `runtimeSignatures: true` +- **THEN** the CLI SHALL emit no entitlement warning + +#### Scenario: Static rules never warn + +- **WHEN** `rule create` writes an `sg` or `vale` rule +- **THEN** the CLI SHALL emit no entitlement warning diff --git a/openspec/specs/cli-rule-reconciliation/spec.md b/openspec/specs/cli-rule-reconciliation/spec.md index d12c48d9..6e373e95 100644 --- a/openspec/specs/cli-rule-reconciliation/spec.md +++ b/openspec/specs/cli-rule-reconciliation/spec.md @@ -278,3 +278,34 @@ upgrading is regeneration and SHALL remain an explicit action. - **WHEN** a newer generation of the same rule exists server-side - **THEN** re-fetch SHALL still return the bytes matching the signature the client reported + +### Requirement: The CLI reads the reconcile entitlement object + +The CLI SHALL read the optional `entitlement` object on a successful reconcile response. It SHALL treat the organization as unentitled only when `entitlement.runtimeSignatures` is exactly `false`; an absent, malformed, or `true` value SHALL be treated as no entitlement outcome, leaving the four buckets to drive execution exactly as before. For an unentitled response the CLI SHALL read `reason`, `upgradeUrl`, and `withheld` (a list of `{ ruleId, file }`), SHALL drop a `withheld` entry without a string `file`, and SHALL surface `upgradeUrl` only when it is an absolute `https:` URL. + +`entitlement.withheld` SHALL be a disposition distinct from the four buckets: a withheld file SHALL NOT be executed, SHALL NOT be surfaced as tamper, drift, or never-issued, and SHALL NOT be sent to restore. The CLI SHALL match a withheld entry to a local runtime rule by the `file` it reported, since the entry carries no signature. A withheld entry that matches no reported file SHALL still count as withheld for the purpose of the exit code. + +#### Scenario: Absent entitlement changes nothing + +- **WHEN** a reconcile response carries no `entitlement` object +- **THEN** the CLI SHALL drive execution from `run`, `unsafe`, `unknown`, and `missing` exactly as before + +#### Scenario: Entitled response changes nothing + +- **WHEN** a reconcile response carries `entitlement: { runtimeSignatures: true }` +- **THEN** the CLI SHALL drive execution from the four buckets exactly as before + +#### Scenario: Withheld is matched by reported path + +- **WHEN** `entitlement.withheld` lists `{ ruleId, file }` and `file` equals the path the CLI reported for a runtime rule's `check.ts` +- **THEN** the CLI SHALL classify that rule as withheld for entitlement and SHALL NOT execute it + +#### Scenario: Withheld is not repaired + +- **WHEN** a runtime rule is withheld for entitlement +- **THEN** the CLI SHALL NOT request a restore for it + +#### Scenario: A malformed upgrade URL is not shown + +- **WHEN** `entitlement.upgradeUrl` is not an absolute `https:` URL +- **THEN** the CLI SHALL omit it from human and `--json` output diff --git a/packages/cli/src/agent/check.md b/packages/cli/src/agent/check.md index 5f0129d4..cccdfcdb 100644 --- a/packages/cli/src/agent/check.md +++ b/packages/cli/src/agent/check.md @@ -1,4 +1,4 @@ -# Topic: check (CLI v%(CLI_VERSION)s / topic v2) +# Topic: check (CLI v%(CLI_VERSION)s / topic v3) ## Goal Run the applicable rules against the codebase and report matches. Two @@ -28,6 +28,9 @@ in CI (diff-only scan), or after rule create/improve to validate. - **Logged in** (token or API key): each rule's `check.ts` is reconciled against the Taskless service; rules the server blessed (`run`) execute, and the rest are withheld and reported (advisory). + **One exception fails the run:** if the organization's plan does not + include runtime rules, the service withholds them for the plan and + `check` exits 1 even with no findings. See "Withheld for the plan". - **Logged out, `--anonymous`, no GitHub remote, or service unavailable**, runtime rules are **skipped** (reported, never run). Static rules still run. @@ -36,14 +39,40 @@ in CI (diff-only scan), or after rule create/improve to validate. This is the only way to run runtime rules unverified. Notices about skipped/withheld runtime rules are human-readable stderr -only: they never change the exit code. Under `--json` they do NOT appear +only, and apart from a withhold for the plan they never change the exit +code. Under `--json` they do NOT appear as warnings; instead an additive optional `skipped: [{ rule, reason }]` array is included alongside the unchanged `{ success, results }`. The authoritative allow-list is the server's; the CI backstop (`%(TASKLESS_CLI)s agent ci`) is the enforcement point for runtime rules. +## Withheld for the plan + +When the organization's Taskless plan does not include runtime rules, +reconcile answers normally but declines to run them. `check` then: + +- does not run them, and reports each in `skipped` with the reason + `not included in your Taskless plan` (never as unsafe or drift); +- prints ONE notice naming the rules, the reason code, and the upgrade + URL; +- **exits 1**, and under `--json` sets `success: false` and adds: + ```json + "entitlement": { + "runtimeSignatures": false, + "reason": "RUNTIME_SIGNATURES_NOT_IN_PLAN", + "upgradeUrl": "https://…", + "withheld": [""] + } + ``` + +Report this to the user as a plan problem, not a code problem: the +rules did not stop matching, they stopped running. Do not edit or +delete the rules to make `check` pass, and do not suggest +`--dangerously-run-scripts` as a fix. Show the `upgradeUrl`. An +`entitlement` with an empty `withheld` does not fail the run. + ## Flags -- `--json`: machine output (`{ success, results, skipped? }`). +- `--json`: machine output (`{ success, results, skipped?, entitlement? }`). - `--anonymous`: run only static rules; skip runtime rules. - `--dangerously-run-scripts`: run runtime `check.ts` unverified. - `--timeout `: per-runtime-check wall-clock bound (default 10). @@ -72,7 +101,9 @@ authoritative allow-list is the server's; the CI backstop When runtime rules were present but not run (e.g. logged out), the JSON also carries `"skipped": [{ "rule": "", "reason": "…" }]` alongside `success`/`results`. Surface it so CI can tell that runtime - rules did not execute; it never affects the exit code. + rules did not execute. It never affects the exit code on its own; an + accompanying `entitlement` with a non-empty `withheld` does (see + "Withheld for the plan"). 3. **Parse the JSON output.** Shape: ```json @@ -99,9 +130,10 @@ authoritative allow-list is the server's; the CI backstop 4. **Report findings to the user.** Group by `file`. Show `severity`, `message`, and `ruleId` for each finding; the `range.start` is the - useful line/column to surface. The `success` field reflects only + useful line/column to surface. The `success` field reflects error-severity findings: `success: false` means at least one - `severity: "error"` finding exists (exit code 1); `success: true` + `severity: "error"` finding exists, or runtime rules were withheld + for the plan (check `entitlement`) (exit code 1); `success: true` with a non-empty `results` array means there are only warning/info/hint findings (exit code 0); `success: true` with an empty `results` array means the codebase is clean. Findings are @@ -113,7 +145,8 @@ authoritative allow-list is the server's; the CI backstop - `0`: All checks passed, no rules configured, or all supplied paths missing -- `1`: Errors detected or scan failed +- `1`: Errors detected, scan failed, or runtime rules withheld because + the plan does not include them ## Errors diff --git a/packages/cli/src/agent/ci.md b/packages/cli/src/agent/ci.md index 61ca69e2..29819799 100644 --- a/packages/cli/src/agent/ci.md +++ b/packages/cli/src/agent/ci.md @@ -1,4 +1,4 @@ -# Topic: ci (CLI v%(CLI_VERSION)s / topic v1) +# Topic: ci (CLI v%(CLI_VERSION)s / topic v2) ## Goal Wire `%(TASKLESS_CLI)s check` into the user's existing CI so rules run @@ -69,6 +69,12 @@ Run `%(TASKLESS_CLI)s check`: - "No rules configured" → stop. Fetch `%(TASKLESS_CLI)s agent route`. - Findings → tell the user CI will fail; ask whether to fix, suppress, or proceed knowing the first CI run will be red. +- Runtime rules withheld for the plan (a notice with an upgrade URL, + or `entitlement.withheld` under `--json`) → tell the user CI will + fail until the organization's plan includes runtime rules. This is + not a findings failure and editing the rules will not fix it. Do not + add `--anonymous` or `--dangerously-run-scripts` to the CI command to + get green; both hide that the rules are not running. ### 4. Generate the config diff --git a/packages/cli/src/api/entitlement.ts b/packages/cli/src/api/entitlement.ts new file mode 100644 index 00000000..e8164253 --- /dev/null +++ b/packages/cli/src/api/entitlement.ts @@ -0,0 +1,107 @@ +/** + * The runtime-signatures entitlement the service attaches to reconcile, + * restore, and request retrieval (taskless/taskless#207). + * + * The object is additive: the four reconcile buckets mean what they always + * meant, and a CLI that ignores it never executes a withheld rule. What such a + * CLI does get wrong is the exit code. A withheld rule is absent from `run`, so + * it is skipped like any other, and `check` goes green while the rule it was + * supposed to be running has stopped. Reading this object is what lets `check` + * tell "the server declined to run this for your plan" apart from drift. + */ + +/** A reported file the service withheld for entitlement, not for tampering. */ +export interface WithheldEntry { + ruleId?: string; + file: string; +} + +/** + * An entitlement outcome. Only the unentitled case is represented: an absent, + * malformed, or `runtimeSignatures: true` object normalizes to `undefined`, + * because none of them asks the CLI to do anything different. + */ +export interface Entitlement { + runtimeSignatures: false; + reason?: string; + /** Present only when it parsed as an absolute `https:` URL. */ + upgradeUrl?: string; + /** Always an array; empty on restore/retrieval, which never carry it. */ + withheld: WithheldEntry[]; +} + +/** + * A generated response type that may also carry `entitlement`. + * + * The service adds the field (taskless/taskless#207) before the vendored + * schema does, so the CLI ships assuming it MIGHT be there rather than waiting + * on the deploy. `unknown`, not `Entitlement`: the field is untrusted until it + * has been through {@link parseEntitlement}. When the schema carries it, this + * collapses to the generated type and the wrapper can go. + */ +export type MayCarryEntitlement = T & { entitlement?: unknown }; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** + * An upgrade URL is printed as a link the user is invited to follow, so a value + * that is not an absolute `https:` URL is dropped rather than shown. + */ +function parseUpgradeUrl(value: unknown): string | undefined { + if (typeof value !== "string") return undefined; + try { + return new URL(value).protocol === "https:" ? value : undefined; + } catch { + return undefined; + } +} + +/** + * Normalize an untrusted `entitlement` field. Returns `undefined` unless + * `runtimeSignatures` is exactly `false`, so a service that predates the field + * (or sends `true`) leaves every caller on its existing path. + */ +export function parseEntitlement(value: unknown): Entitlement | undefined { + if (!isRecord(value) || value.runtimeSignatures !== false) return undefined; + + const withheld: WithheldEntry[] = []; + if (Array.isArray(value.withheld)) { + for (const entry of value.withheld) { + // The file is the join key back to a local rule (the entry carries no + // signature), and the schema requires it. An entry without one broke + // that contract and is dropped, not guessed at. An entry that HAS a file + // matching nothing local is kept: it still fails the run. + if (!isRecord(entry) || typeof entry.file !== "string") continue; + withheld.push({ + file: entry.file, + ...(typeof entry.ruleId === "string" ? { ruleId: entry.ruleId } : {}), + }); + } + } + + const upgradeUrl = parseUpgradeUrl(value.upgradeUrl); + return { + runtimeSignatures: false, + ...(typeof value.reason === "string" ? { reason: value.reason } : {}), + ...(upgradeUrl === undefined ? {} : { upgradeUrl }), + withheld, + }; +} + +/** + * Why a runtime rule just written will not run. Shared by `rule create`, + * `rule improve`, and restore, which each name the rule their own way, so the + * explanation cannot drift between them. + */ +export function notRunOnPlanSentence(entitlement: Entitlement): string { + // The URL ends the sentence with no trailing period, so copying it from a + // terminal does not copy a `.` into the address. + return ( + "It will not run: runtime rules are not included in your Taskless plan" + + (entitlement.upgradeUrl === undefined + ? "." + : `. Upgrade at ${entitlement.upgradeUrl}`) + ); +} diff --git a/packages/cli/src/api/reconcile.ts b/packages/cli/src/api/reconcile.ts index 66806741..385a92c5 100644 --- a/packages/cli/src/api/reconcile.ts +++ b/packages/cli/src/api/reconcile.ts @@ -1,4 +1,5 @@ import { getApiBaseUrl } from "./config"; +import { parseEntitlement, type Entitlement } from "./entitlement"; import { CLI_VERSION, CLI_VERSION_HEADER } from "../version"; /** @@ -57,6 +58,11 @@ export interface ReconcileResponse { unsafe: UnsafeEntry[]; unknown: UnknownEntry[]; missing: MissingEntry[]; + /** + * Present only when the service withheld runtime blessing for the plan. + * Absent means the four buckets are the whole answer, as they always were. + */ + entitlement?: Entitlement; } /** @@ -124,7 +130,8 @@ export async function reconcile( return { status: "unavailable", reason: "invalid response body" }; } - const data = body as Partial; + const data = body as Partial>; + const entitlement = parseEntitlement(data.entitlement); return { status: "ok", result: { @@ -132,6 +139,7 @@ export async function reconcile( unsafe: asArray(data.unsafe), unknown: asArray(data.unknown), missing: asArray(data.missing), + ...(entitlement === undefined ? {} : { entitlement }), }, }; } diff --git a/packages/cli/src/api/restore.ts b/packages/cli/src/api/restore.ts index 8c437691..fe2b793f 100644 --- a/packages/cli/src/api/restore.ts +++ b/packages/cli/src/api/restore.ts @@ -1,5 +1,10 @@ import type { paths } from "../generated/api"; import { getApiBaseUrl } from "./config"; +import { + parseEntitlement, + type Entitlement, + type MayCarryEntitlement, +} from "./entitlement"; import { CLI_VERSION, CLI_VERSION_HEADER } from "../version"; /** @@ -36,7 +41,7 @@ export type RestoredRule = NonNullable[number]; * already a safe state. */ export type RestoreOutcome = - | { status: "ok"; rules: RestoredRule[] } + | { status: "ok"; rules: RestoredRule[]; entitlement?: Entitlement } | { status: "unauthorized" } | { status: "unavailable"; reason: string }; @@ -92,10 +97,16 @@ export async function restoreRule( return { status: "unavailable", reason: "invalid response body" }; } - const data = body as Partial; + const data = body as Partial>; const rules = data.rules; if (!Array.isArray(rules)) { return { status: "unavailable", reason: "response carried no `rules`" }; } - return { status: "ok", rules }; + // Absent means entitled, and every service before #207 omits it. + const entitlement = parseEntitlement(data.entitlement); + return { + status: "ok", + rules, + ...(entitlement === undefined ? {} : { entitlement }), + }; } diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts index 52383c46..844a0c1f 100644 --- a/packages/cli/src/commands/check.ts +++ b/packages/cli/src/commands/check.ts @@ -413,7 +413,17 @@ export const checkCommand = defineCommand({ // Computed by `runEngines`, not here: the exit code is a fact about a // completed dispatch, and an engine failure has to fail the check even // with no findings. - const { exitCode } = dispatched; + // + // With one exception decided here: a runtime rule the service withheld + // for the plan fails the run whatever the scan found. Every other skip + // leaves the exit code alone because the CLI could not ask; this one + // is the answer to asking, and a green run would say the rule is still + // protecting the repository when it has stopped running. + const withheldForPlan = (plan.entitlement?.withheld.length ?? 0) > 0; + const exitCode = + dispatched.exitCode === 0 && withheldForPlan + ? 1 + : dispatched.exitCode; if (args.json) { const output = checkOutputSchema.parse({ @@ -429,6 +439,9 @@ export const checkCommand = defineCommand({ // channel a CI run reads dropped the entire output of the feature // whose whole purpose is explaining a rule that did not run. ...(runNotices.length > 0 ? { notices: runNotices } : {}), + ...(plan.entitlement === undefined + ? {} + : { entitlement: plan.entitlement }), }); console.log(JSON.stringify(output)); } else { diff --git a/packages/cli/src/commands/rules.ts b/packages/cli/src/commands/rules.ts index 37b491b6..540f8b79 100644 --- a/packages/cli/src/commands/rules.ts +++ b/packages/cli/src/commands/rules.ts @@ -13,6 +13,11 @@ import { isSingleContentRule, type GeneratedRule, } from "../api/rules"; +import { + notRunOnPlanSentence, + parseEntitlement, + type MayCarryEntitlement, +} from "../api/entitlement"; import { writeRuleFile, writeRuleTestFile, @@ -20,6 +25,7 @@ import { readRuleMetaFile, deleteRuleFiles, } from "../rules/files"; +import { resolveIngestEngine } from "../rules/engines"; import { RULES_DIRECTORY } from "../rules/layout"; import { unsupportedMessage } from "../rules/unsupported"; import { @@ -35,6 +41,25 @@ import { getTelemetry } from "../telemetry"; import { CLIError } from "../util/cli-error"; import { type CLIErrorCode, writeJsonError } from "../types/errors"; +/** + * The warning for a runtime rule written under a plan that will not run it, or + * `undefined` when there is nothing to say. + * + * The rule is still written: it is the organization's rule, and it runs again + * the moment the plan allows. What must not happen is the author finishing + * `rule create` believing it is live. Static rules never warn. + */ +function notRunOnPlanNotice( + status: MayCarryEntitlement, + rule: unknown, + ruleFile: string +): string | undefined { + const entitlement = parseEntitlement(status.entitlement); + if (entitlement === undefined) return undefined; + if (resolveIngestEngine(rule) !== "runtime") return undefined; + return `${ruleFile} was written. ${notRunOnPlanSentence(entitlement)}`; +} + /** Format today's date as YYYYMMDD */ function getTimestamp(): string { const now = new Date(); @@ -276,6 +301,11 @@ const createCommand = defineCommand({ if (!args.json) console.error(`Warning: ${message}`); }); writtenFiles.push(ruleFile); + const planWarning = notRunOnPlanNotice(status, rule, ruleFile); + if (planWarning !== undefined) { + notices.push(planWarning); + if (!args.json) console.error(`Warning: ${planWarning}`); + } // See `fileSetTestsFieldError` for why this is a guard rather // than a silent drop. @@ -547,6 +577,11 @@ const improveCommand = defineCommand({ if (!args.json) console.error(`Warning: ${message}`); }); writtenFiles.push(ruleFile); + const planWarning = notRunOnPlanNotice(status, rule, ruleFile); + if (planWarning !== undefined) { + notices.push(planWarning); + if (!args.json) console.error(`Warning: ${planWarning}`); + } // See `fileSetTestsFieldError` for why this is a guard rather // than a silent drop. diff --git a/packages/cli/src/rules/runtime/plan.ts b/packages/cli/src/rules/runtime/plan.ts index 96d6ce23..0113a88b 100644 --- a/packages/cli/src/rules/runtime/plan.ts +++ b/packages/cli/src/rules/runtime/plan.ts @@ -4,6 +4,7 @@ import { resolveRepositoryUrl } from "../../util/git-remote"; import { getCliPrefix } from "../../util/package-manager"; import { reconcile } from "../../api/reconcile"; import type { ReconcileResponse } from "../../api/reconcile"; +import { notRunOnPlanSentence } from "../../api/entitlement"; import { restoreRule } from "../../api/restore"; import { writeRuleFile } from "../files"; import { PurgeIncompleteError } from "../deliver"; @@ -12,6 +13,7 @@ import { RUN_SCRIPTS_WARNING } from "./harness"; import { type RuntimeRule } from "./discover"; import { materializeRuntimeRules, + reportedCheckPath, reportRuntimeChecks, selectBlessedRuntimeRules, signRuntimeChecks, @@ -46,6 +48,24 @@ export interface SkippedRuntimeRule { reason: string; } +/** + * The service declined to run runtime rules for this organization's plan. + * + * Unlike every other skip, this one fails `check`. The degrade paths skip + * because the CLI could not ask; this skip is the answer to a question it did + * ask, and it will be the same answer on every run until someone acts on it. + */ +export interface PlanEntitlement { + runtimeSignatures: false; + reason?: string; + upgradeUrl?: string; + /** + * Local rule names the service withheld, plus the reported path of any + * withheld entry that matched no local rule. Non-empty means `check` fails. + */ + withheld: string[]; +} + /** The runtime-execution plan resolved from auth state and flags. */ export interface RuntimePlan { /** Rules to execute — materialized when gated, live under `--dangerously-run-scripts`. */ @@ -54,8 +74,13 @@ export interface RuntimePlan { skipped: SkippedRuntimeRule[]; /** Human-only notices about the runtime disposition. */ notices: string[]; + /** Present only when reconcile answered for a plan without runtime signatures. */ + entitlement?: PlanEntitlement; } +/** The skip reason for a rule withheld because the plan lacks runtime rules. */ +export const NOT_IN_PLAN_REASON = "not included in your Taskless plan"; + /** Skip every runtime rule with a shared reason (an unverified path). */ function skipAllRuntime(rules: RuntimeRule[], reason: string): RuntimePlan { return { @@ -179,6 +204,17 @@ export async function planRuntime( signed, outcome.result.run ); + // Joined by reported path, since a withheld entry carries no signature. A + // withheld rule is split out of the generic "not blessed" skips so it is + // never described as drift: nothing about its bytes is wrong. + const entitlement = outcome.result.entitlement; + const withheldFiles = new Set( + (entitlement?.withheld ?? []).map((entry) => entry.file) + ); + const planWithheld = withheld.filter((rule) => + withheldFiles.has(reportedCheckPath(cwd, rule)) + ); + const notBlessed = withheld.filter((rule) => !planWithheld.includes(rule)); let execute: RuntimeRule[] = []; try { execute = @@ -195,9 +231,22 @@ export async function planRuntime( // below whether or not its bytes were just restored. Fetching code and // executing it in the same pass that discovered the drift would move the // gate, and the gate is the point. + // + // A file withheld for the plan is never sent to restore. The service keeps + // it out of `unsafe` and `missing` already; this holds if it ever does not, + // because restoring bytes the plan will not run fixes nothing and says the + // opposite. const repair = await repairWithheldRules(cwd, token, { repositoryUrl, - result: outcome.result, + result: { + ...outcome.result, + unsafe: outcome.result.unsafe.filter( + (entry) => !withheldFiles.has(entry.file) + ), + missing: outcome.result.missing.filter( + (entry) => !withheldFiles.has(entry.file) + ), + }, }); // A rule can be blessed and then vanish before it is executed. `execute` is @@ -214,20 +263,77 @@ export async function planRuntime( // independently of which specific route caused it. const droppedSkips = accountForDroppedRules(blessed, execute); + const planEntitlement = + entitlement === undefined + ? undefined + : summarizeEntitlement(cwd, entitlement, planWithheld); + return { execute, skipped: [ ...unreadableSkips, - ...withheld.map((rule) => ({ + ...notBlessed.map((rule) => ({ rule: rule.name, reason: "not blessed by the server (unsafe / unknown / drift)", })), + ...planWithheld.map((rule) => ({ + rule: rule.name, + reason: NOT_IN_PLAN_REASON, + })), ...droppedSkips, ], - notices: repair.notices, + notices: [ + ...(planEntitlement === undefined || planEntitlement.withheld.length === 0 + ? [] + : [withheldNotice(planEntitlement)]), + ...repair.notices, + ], + ...(planEntitlement === undefined ? {} : { entitlement: planEntitlement }), }; } +/** + * Name what the service withheld, locally where possible. + * + * A withheld entry whose file matches no local rule is kept by its reported + * path rather than dropped. The service said something will not run, and the + * CLI failing to attribute it is not a reason for the run to go green. + */ +function summarizeEntitlement( + cwd: string, + entitlement: NonNullable, + planWithheld: RuntimeRule[] +): PlanEntitlement { + const matched = new Set( + planWithheld.map((rule) => reportedCheckPath(cwd, rule)) + ); + const unmatched = entitlement.withheld + .map((entry) => entry.file) + .filter((file) => !matched.has(file)); + return { + runtimeSignatures: false, + ...(entitlement.reason === undefined ? {} : { reason: entitlement.reason }), + ...(entitlement.upgradeUrl === undefined + ? {} + : { upgradeUrl: entitlement.upgradeUrl }), + withheld: [...planWithheld.map((rule) => rule.name), ...unmatched], + }; +} + +/** The one notice a withheld run prints, so the upgrade URL appears once. */ +function withheldNotice(entitlement: PlanEntitlement): string { + const count = entitlement.withheld.length; + return ( + `${String(count)} runtime ${count === 1 ? "rule was" : "rules were"} ` + + `withheld because runtime rules are not included in your Taskless plan` + + (entitlement.reason === undefined ? "" : ` (${entitlement.reason})`) + + `: ${entitlement.withheld.join(", ")}. \`check\` fails until they can run` + + (entitlement.upgradeUrl === undefined + ? "." + : `. Upgrade at ${entitlement.upgradeUrl}`) + ); +} + /** * Act on the verdicts `check` used to parse and discard. * @@ -341,6 +447,16 @@ async function repairWithheldRules( notices.push(`${target.file} could not be written (${message}).`); continue; } + // The usual notice below promises the next `check` blesses the rule, which + // a plan without runtime signatures will not do. The bytes are still the + // right bytes and are still written; only the promise is withdrawn. + if (outcome.entitlement !== undefined) { + notices.push( + `${target.file} was restored with the bytes the service blessed. ` + + notRunOnPlanSentence(outcome.entitlement) + ); + continue; + } // Now says what the DIRECTORY contains, not just what was written. The // delivered set is authoritative (see `writeDeliveredFileSet`), so a file // the set does not name — a stray capture beside the rule, which reconcile diff --git a/packages/cli/src/rules/runtime/run-set.ts b/packages/cli/src/rules/runtime/run-set.ts index ac64f6d9..b711aaeb 100644 --- a/packages/cli/src/rules/runtime/run-set.ts +++ b/packages/cli/src/rules/runtime/run-set.ts @@ -47,14 +47,23 @@ export async function signRuntimeChecks( return { signed, unreadable }; } +/** + * The path a rule's `check.ts` is reported under. One function, because + * `entitlement.withheld` carries no signature and is joined back to local rules + * by this path: a second spelling of it would silently match nothing. + */ +export function reportedCheckPath(cwd: string, rule: RuntimeRule): string { + // Reconcile paths are repo-relative POSIX; normalize Windows separators. + return relative(cwd, rule.checkFile).split(sep).join("/"); +} + /** Map signed runtime rules to the reconcile report (`check.ts` path + signature). */ export function reportRuntimeChecks( cwd: string, signed: SignedRuntimeRule[] ): ReportedFile[] { return signed.map(({ rule, signature }) => ({ - // Reconcile paths are repo-relative POSIX; normalize Windows separators. - file: relative(cwd, rule.checkFile).split(sep).join("/"), + file: reportedCheckPath(cwd, rule), signature, })); } diff --git a/packages/cli/src/schemas/check.ts b/packages/cli/src/schemas/check.ts index 68afea9f..fde389bf 100644 --- a/packages/cli/src/schemas/check.ts +++ b/packages/cli/src/schemas/check.ts @@ -49,6 +49,28 @@ export const outputSchema = z.object({ .array(z.string()) .optional() .describe("Advisory messages: engines that could not run"), + // Present only when reconcile answered for a plan without runtime rules. A + // non-empty `withheld` is why `success` is false on a run with no findings, + // and the one field that tells a CI job the fix is the plan, not the code. + entitlement: z + .object({ + runtimeSignatures: z.literal(false), + reason: z + .string() + .optional() + .describe( + "Machine-readable reason, e.g. RUNTIME_SIGNATURES_NOT_IN_PLAN" + ), + upgradeUrl: z + .string() + .optional() + .describe("Where the organization can upgrade its plan"), + withheld: z + .array(z.string()) + .describe("Runtime rules the service declined to run for this plan"), + }) + .optional() + .describe("Runtime rules withheld because the plan does not include them"), }); /** Error schema for `taskless check --json` on failure */ diff --git a/packages/cli/test/entitlement.test.ts b/packages/cli/test/entitlement.test.ts new file mode 100644 index 00000000..22536907 --- /dev/null +++ b/packages/cli/test/entitlement.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from "vitest"; + +import { parseEntitlement } from "../src/api/entitlement"; + +const UPGRADE = "https://app.taskless.io/o/acme/upgrade?from=reconcile"; + +describe("parseEntitlement", () => { + it("is undefined when the field is absent, so an older service changes nothing", () => { + const body: { entitlement?: unknown } = {}; + expect(parseEntitlement(body.entitlement)).toBeUndefined(); + }); + + it("is undefined for an entitled organization", () => { + expect(parseEntitlement({ runtimeSignatures: true })).toBeUndefined(); + }); + + it("is undefined unless runtimeSignatures is exactly false", () => { + for (const value of [null, "false", 0, [], { runtimeSignatures: "no" }]) { + expect(parseEntitlement(value)).toBeUndefined(); + } + }); + + it("reads reason, upgradeUrl, and withheld for an unentitled organization", () => { + expect( + parseEntitlement({ + runtimeSignatures: false, + reason: "RUNTIME_SIGNATURES_NOT_IN_PLAN", + upgradeUrl: UPGRADE, + withheld: [ + { ruleId: "r-1", file: ".taskless/rules/runtime/a/check.ts" }, + ], + }) + ).toEqual({ + runtimeSignatures: false, + reason: "RUNTIME_SIGNATURES_NOT_IN_PLAN", + upgradeUrl: UPGRADE, + withheld: [{ ruleId: "r-1", file: ".taskless/rules/runtime/a/check.ts" }], + }); + }); + + it("defaults withheld to empty, as restore and retrieval send it", () => { + expect(parseEntitlement({ runtimeSignatures: false })).toEqual({ + runtimeSignatures: false, + withheld: [], + }); + }); + + it("drops a withheld entry without a string file, and keeps one without a ruleId", () => { + const parsed = parseEntitlement({ + runtimeSignatures: false, + withheld: [ + { ruleId: "r-1" }, + "a/check.ts", + null, + { file: 42 }, + { file: "b/check.ts" }, + ], + }); + expect(parsed?.withheld).toEqual([{ file: "b/check.ts" }]); + }); + + it("omits an upgradeUrl that is not an absolute https URL", () => { + for (const upgradeUrl of [ + "/o/acme/upgrade", + "http://app.taskless.io/upgrade", + "javascript:alert(1)", + "not a url", + 42, + ]) { + const parsed = parseEntitlement({ runtimeSignatures: false, upgradeUrl }); + expect(parsed).toBeDefined(); + expect(parsed).not.toHaveProperty("upgradeUrl"); + } + }); +}); diff --git a/packages/cli/test/repair-integration.test.ts b/packages/cli/test/repair-integration.test.ts index 72340082..926b4207 100644 --- a/packages/cli/test/repair-integration.test.ts +++ b/packages/cli/test/repair-integration.test.ts @@ -247,6 +247,69 @@ describe("repairing a drifted runtime rule, end to end", () => { } }); + it("does not promise blessing when the plan will not run the restored rule", async () => { + // The ordinary notice says the next `check` blesses the repaired bytes. + // Under a plan without runtime signatures it will not, so the restore + // still writes the blessed bytes and withdraws only the promise. + const upgradeUrl = "https://app.taskless.io/o/acme/upgrade?from=restore"; + const blessed = await canonicalHash(BLESSED); + const restoreBody = (entitlement: unknown) => ({ + statusCode: 200, + body: { + ruleId: "demo", + entitlement, + rules: [ + { + id: "demo", + engine: "runtime", + files: [ + { path: "check.ts", content: BLESSED }, + { path: "captures/logs.yml", content: CAPTURE }, + ], + signature: blessed, + }, + ], + }, + }); + + const mock = await startMock({ + reconcile: driftedReconcile(blessed), + restore: () => restoreBody({ runtimeSignatures: false, upgradeUrl }), + }); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: mock.apiUrl, + }); + const notices = (envelope(stdout).notices ?? []).join("\n"); + expect(notices).toContain(`${REPORTED} was restored`); + expect(notices).toMatch(/will not run/); + expect(notices).toContain(upgradeUrl); + expect(notices).not.toMatch(/next `check`/); + await expect(readFile(checkFile, "utf8")).resolves.toBe(BLESSED); + } finally { + await mock.close(); + } + + // An entitled response keeps the ordinary notice and warns about nothing. + await writeFile(checkFile, DRIFTED, "utf8"); + const entitled = await startMock({ + reconcile: driftedReconcile(blessed), + restore: () => restoreBody({ runtimeSignatures: true }), + }); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: entitled.apiUrl, + }); + const notices = (envelope(stdout).notices ?? []).join("\n"); + expect(notices).toMatch(/next `check`/); + expect(notices).not.toMatch(/will not run/); + } finally { + await entitled.close(); + } + }); + it("leaves the rule directory holding exactly the blessed set, minus fixtures", async () => { // #233. Repair runs BECAUSE the directory's trustworthiness is in // question, and only `check.ts` is signed — so a stray capture beside the diff --git a/packages/cli/test/rule-create-entitlement.test.ts b/packages/cli/test/rule-create-entitlement.test.ts new file mode 100644 index 00000000..dcd00cd0 --- /dev/null +++ b/packages/cli/test/rule-create-entitlement.test.ts @@ -0,0 +1,208 @@ +import { execFileSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + afterEach, + beforeEach, + describe, + expect, + it, + vi, + type MockInstance, +} from "vitest"; + +import { runCommand } from "citty"; + +import { ruleCommand } from "../src/commands/rules"; + +/** + * A runtime rule written under a plan without runtime signatures is still + * written, and says it will not run. Driven through the real command, as + * `rule-guard-json-envelope.test.ts` is, because the warning's whole job is to + * reach the `--json` envelope an unattended author reads. + */ + +const UPGRADE = "https://app.taskless.io/o/acme/upgrade?from=delivery"; + +const CAPTURE = [ + "id: logs-abc12345", + "language: typescript", + "rule:", + " pattern: console.log($A)", + "metadata:", + " taskless:", + " version: 1", + " kind: runtime", + " name: logs", + " check: check.ts", + " match: anchor", + "", +].join("\n"); + +const runtimeRule = { + id: "plan-runtime-rule", + engine: "runtime", + files: [ + { path: "check.ts", content: "export default async () => [];\n" }, + { path: "captures/logs.yml", content: CAPTURE }, + ], +}; + +const staticRule = { + id: "plan-static-rule", + engine: "sg", + files: [ + { + path: "plan-static-rule.yml", + content: + "id: plan-static-rule\nlanguage: TypeScript\nrule:\n pattern: foo\n", + }, + ], +}; + +describe("rule create/improve: a runtime rule the plan will not run", () => { + let cwd: string; + let logSpy: MockInstance<(...data: unknown[]) => void>; + + const requestId = "33333333-3333-3333-3333-333333333333"; + const iterateRequestId = "44444444-4444-4444-4444-444444444444"; + + function stubFetch(status: Record): void { + vi.stubGlobal( + "fetch", + vi.fn((input: string | URL | Request, init?: RequestInit) => { + const url = new URL( + typeof input === "string" + ? input + : input instanceof URL + ? input.href + : input.url + ); + const method = ( + init?.method ?? (input instanceof Request ? input.method : "GET") + ).toUpperCase(); + const { pathname } = url; + if (pathname === "/cli/api/whoami") { + return Response.json({}, { status: 500 }); + } + if (method === "POST" && pathname === "/cli/api/request") { + return Response.json({ requestId }, { status: 200 }); + } + if (method === "POST" && pathname.endsWith("/iterate")) { + return Response.json({ requestId: iterateRequestId }); + } + if (method === "GET" && pathname.startsWith("/cli/api/request/")) { + return Response.json({ status: "generated", ...status }); + } + throw new Error(`unexpected ${method} ${pathname}`); + }) + ); + } + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-rule-plan-")); + await mkdir(join(cwd, ".taskless"), { recursive: true }); + await writeFile( + join(cwd, ".taskless", "taskless.json"), + JSON.stringify({ + version: "2026-03-03", + orgId: 123, + repositoryUrl: "https://github.com/test/test", + }) + ); + execFileSync("git", ["init"], { cwd }); + execFileSync( + "git", + ["remote", "add", "origin", "https://github.com/test/test.git"], + { cwd } + ); + process.env.TASKLESS_TOKEN = "test-token"; + process.env.TASKLESS_API_URL = "https://example.invalid/cli"; + logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + vi.spyOn(console, "error").mockImplementation(() => {}); + vi.stubGlobal( + "setTimeout", + (function_: (...arguments_: unknown[]) => void) => { + function_(); + return 0 as unknown as NodeJS.Timeout; + } + ); + }); + + afterEach(async () => { + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + delete process.env.TASKLESS_TOKEN; + delete process.env.TASKLESS_API_URL; + await rm(cwd, { recursive: true, force: true }); + }); + + /** The plan warnings in the last `--json` envelope. */ + function planWarnings(): string[] { + const last = logSpy.mock.calls.at(-1); + if (!last) throw new Error("console.log was never called"); + const envelope = JSON.parse(String(last[0])) as { notices?: string[] }; + return (envelope.notices ?? []).filter((n) => n.includes("will not run")); + } + + async function create(): Promise { + const requestFile = join(cwd, "request.json"); + await writeFile(requestFile, JSON.stringify({ prompt: "add a rule" })); + await runCommand(ruleCommand, { + rawArgs: ["create", "--from", requestFile, "--json", "-d", cwd], + }); + } + + it("rule create writes the rule and warns under --json", async () => { + stubFetch({ + rules: [runtimeRule], + entitlement: { runtimeSignatures: false, upgradeUrl: UPGRADE }, + }); + await create(); + + const warnings = planWarnings(); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain("plan-runtime-rule"); + expect(warnings[0]).toContain(UPGRADE); + expect( + existsSync( + join(cwd, ".taskless", "rules", "runtime", "plan-runtime-rule") + ) + ).toBe(true); + }); + + it("rule improve warns the same way", async () => { + stubFetch({ + rules: [runtimeRule], + entitlement: { runtimeSignatures: false }, + }); + const requestFile = join(cwd, "improve.json"); + await writeFile( + requestFile, + JSON.stringify({ ruleId: "plan-runtime-rule", guidance: "tighten" }) + ); + await runCommand(ruleCommand, { + rawArgs: ["improve", "--from", requestFile, "--json", "-d", cwd], + }); + expect(planWarnings()).toHaveLength(1); + }); + + it("an entitled or legacy response does not warn", async () => { + for (const entitlement of [undefined, { runtimeSignatures: true }]) { + stubFetch({ rules: [runtimeRule], entitlement }); + await create(); + expect(planWarnings()).toEqual([]); + } + }); + + it("a static rule never warns, whatever the plan", async () => { + stubFetch({ + rules: [staticRule], + entitlement: { runtimeSignatures: false, upgradeUrl: UPGRADE }, + }); + await create(); + expect(planWarnings()).toEqual([]); + }); +}); diff --git a/packages/cli/test/runtime-check.test.ts b/packages/cli/test/runtime-check.test.ts index f06f0294..167e9cbe 100644 --- a/packages/cli/test/runtime-check.test.ts +++ b/packages/cli/test/runtime-check.test.ts @@ -138,6 +138,38 @@ const RUNTIME_CHECK = `export default async function (root, matches) { const CHECK_REPORT_PATH = ".taskless/runtime/rules/demo/check.ts"; +const UPGRADE_URL = "https://app.taskless.io/o/acme/upgrade?from=reconcile"; + +/** The `--json` line with the entitlement field this suite asserts on. */ +function parseEntitlementJson(stdout: string): ReturnType & { + entitlement?: { + runtimeSignatures: false; + reason?: string; + upgradeUrl?: string; + withheld: string[]; + }; +} { + return parseJson(stdout) as ReturnType; +} + +/** A reconcile body withholding the reported files ending in `endsWith`. */ +function withholding(request: ReconcileRequestBody, ...endsWith: string[]) { + return { + run: [], + unsafe: [], + unknown: [], + missing: [], + entitlement: { + runtimeSignatures: false, + reason: "RUNTIME_SIGNATURES_NOT_IN_PLAN", + upgradeUrl: UPGRADE_URL, + withheld: request.files + .filter((f) => endsWith.some((suffix) => f.file.endsWith(suffix))) + .map((f) => ({ ruleId: "r", file: f.file })), + }, + }; +} + describe("check: static vs runtime dispatch", () => { let directory: string; @@ -345,6 +377,208 @@ describe("check: static vs runtime dispatch", () => { } }); + it("withheld for the plan: fails the run and names the cause, not drift", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: withholding(request, "demo/check.ts"), + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseEntitlementJson(stdout); + // The only finding is a warning, so this is the withhold alone failing. + expect(exitCode).toBe(1); + expect(output.success).toBe(false); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + false + ); + const skip = output.skipped?.find((s) => s.rule === "demo"); + expect(skip?.reason).toBe("not included in your Taskless plan"); + expect(skip?.reason).not.toMatch(/unsafe|unknown|drift/); + expect(output.entitlement).toEqual({ + runtimeSignatures: false, + reason: "RUNTIME_SIGNATURES_NOT_IN_PLAN", + upgradeUrl: UPGRADE_URL, + withheld: ["demo"], + }); + } finally { + await server.close(); + } + }); + + it("withheld for the plan, human output: one notice carries the upgrade URL", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: withholding(request, "demo/check.ts"), + })); + try { + const { stderr, exitCode } = await runCli(["check", "-d", directory], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + expect(exitCode).toBe(1); + expect(stderr.split(UPGRADE_URL)).toHaveLength(2); + expect(stderr).toContain("RUNTIME_SIGNATURES_NOT_IN_PLAN"); + } finally { + await server.close(); + } + }); + + it("blessed and withheld together: the blessed rule runs and the run still fails", async () => { + const other = join(directory, ".taskless", "runtime", "rules", "other"); + await mkdir(other, { recursive: true }); + await writeFile(join(other, "logs.yml"), RUNTIME_CAPTURE, "utf8"); + await writeFile(join(other, "check.ts"), RUNTIME_CHECK + "// other\n"); + + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + ...withholding(request, "other/check.ts"), + run: [ + { + ruleId: "demo", + file: CHECK_REPORT_PATH, + signature: sig(request, "demo/check.ts"), + }, + ], + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseEntitlementJson(stdout); + expect(exitCode).toBe(1); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + true + ); + expect(output.entitlement?.withheld).toEqual(["other"]); + } finally { + await server.close(); + } + }); + + it("a withheld file matching no local rule still fails, named by its path", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { + run: [], + unsafe: [], + unknown: [], + missing: [], + entitlement: { + runtimeSignatures: false, + withheld: [{ ruleId: "r", file: "elsewhere/check.ts" }], + }, + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseEntitlementJson(stdout); + expect(exitCode).toBe(1); + expect(output.entitlement?.withheld).toEqual(["elsewhere/check.ts"]); + // `demo` was reported and not withheld, so it keeps the ordinary reason. + expect(output.skipped?.find((s) => s.rule === "demo")?.reason).toMatch( + /not blessed/ + ); + } finally { + await server.close(); + } + }); + + it("a file withheld for the plan is never sent to restore", async () => { + // The service keeps withheld files out of `unsafe`; this is the guard for + // the day it does not. Restoring bytes the plan will not run fixes nothing. + const server = await startMockServer((request) => { + const body = withholding(request, "demo/check.ts"); + const file = body.entitlement.withheld[0]!.file; + return { + statusCode: 200, + body: { + ...body, + unsafe: [ + { ruleId: "r", file, expected: "1;h=sha-256;d=00", got: "x" }, + ], + }, + }; + }); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + const output = JSON.parse( + stdout + .trim() + .split("\n") + .findLast((l) => l.startsWith("{")) ?? "{}" + ) as { notices?: string[] }; + expect( + (output.notices ?? []).some((notice) => /restor/.test(notice)) + ).toBe(false); + } finally { + await server.close(); + } + }); + + it("unentitled with nothing withheld: exit 0, entitlement still reported", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { + run: [], + unsafe: [], + unknown: [], + missing: [], + entitlement: { runtimeSignatures: false, withheld: [] }, + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseEntitlementJson(stdout); + expect(exitCode).toBe(0); + expect(output.success).toBe(true); + expect(output.entitlement).toEqual({ + runtimeSignatures: false, + withheld: [], + }); + } finally { + await server.close(); + } + }); + + it("entitled or legacy responses are unchanged: exit 0, no entitlement field", async () => { + for (const entitlement of [undefined, { runtimeSignatures: true }]) { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { run: [], unsafe: [], unknown: [], missing: [], entitlement }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseEntitlementJson(stdout); + expect(exitCode).toBe(0); + expect(output).not.toHaveProperty("entitlement"); + expect(output.skipped?.find((s) => s.rule === "demo")?.reason).toMatch( + /not blessed/ + ); + } finally { + await server.close(); + } + } + }); + it("declares the CLI version via the x-taskless-cli-version header", async () => { const server = await startMockServer(() => ({ statusCode: 200,