fix(service-messaging): reap once per dispatcher tick, back off while idle, wake on emit

.changeset/17610-notification-dispatcher-idle-cost.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/service-messaging': minor
+---
+
+`NotificationDispatcher` reaps once per tick instead of once per claim, backs off while the outbox is idle, and `emit()` wakes it (#17610)
+
+**What an idle dispatcher cost.** Against an EMPTY `sys_notification_delivery` outbox every tick walked `partitionCount` partitions (default 8) and ran `claim()` and `claimDigest()` in each — and each of those opened with the environment-wide visibility-timeout reap before its candidate SELECT. Measured on a real `ObjectQL` + `SqlDriver`: **32 statements a tick, 16 of them the identical reap UPDATE**, on a fixed 500 ms interval that never let up, one loop per warm kernel. On remote Turso every statement is an HTTP round trip.
+
+**Now:**
+
+- **The reap runs once per tick**, before any claim — an idle tick is `1 + 2 × partitionCount` = 17 statements. Its predicate names no partition, so one run returns every claim that had expired when the tick began; a claim that expires during the tick is returned by the next one. A crashed node's `in_flight` rows are still recovered within one tick of `claimTtlMs` passing, and a claim is still never re-taken before its TTL.
+- **The loop backs off while idle.** Every tick that claims nothing doubles the delay to the next, from `intervalMs` up to `maxIdleIntervalMs` (default 30 s; `MessagingServicePlugin` option `dispatchMaxIdleIntervalMs`). A tick that claims work snaps back to `intervalMs`. With the defaults, ten idle minutes are 24 ticks instead of 1,201.
+- **`emit()` wakes the dispatcher.** `MessagingService.setOutbox(outbox, { onEnqueued })` fires once per `emit()` that enqueued at least one delivery; the plugin points it at the new `NotificationDispatcher.wake()`, which ticks immediately — or once more, right after a tick already in flight.
+
+**Latency bound.** A notification emitted in the process that runs the dispatcher goes out on the tick `wake()` starts, no later than before. While idle, work nobody announces is noticed within one backed-off interval, at most `maxIdleIntervalMs` (30 s by default): a deferred delivery coming due (retry schedule, quiet hours, digest window), a row enqueued by a process that does not run this dispatcher, and a crashed node's expired claim (recovered within `claimTtlMs` + `maxIdleIntervalMs`). Set `dispatchMaxIdleIntervalMs` to `dispatchIntervalMs` to keep the fixed interval.
+
+**Contract additions — all optional, nothing to change on upgrade.** `INotificationOutbox` gains an optional `reap(opts: ReapOptions)` — the visibility-timeout recovery `claim()` / `claimDigest()` already open with, as a method of its own — and `ClaimOptions` gains an optional `skipReap`. Both built-in stores (`SqlNotificationOutbox`, `MemoryNotificationOutbox`) implement them. A custom outbox without `reap()` keeps working as it is: the dispatcher probes for the method and, when it is absent, lets each claim reap as before — correct, at the old per-claim cost; implementing `reap()` and honouring `skipReap` is what earns the once-per-tick cost. Direct callers of `claim()` / `claimDigest()` are unaffected: without `skipReap` they reap exactly as before. Also new: `NotificationDispatcher.wake()`, the dispatcher's `maxIdleIntervalMs` option, and `MessagingService.setOutbox`'s optional second argument.

content/docs/permissions/tenant-audit-census.mdx

@@ -98,7 +98,7 @@ are reported as `undecidable` rather than assumed either way.
 
 The same holds twice over for the context. An options argument spelled as a
 literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
-forwarding shim cannot, and **67 of the 223 sites are spelled that way**. A
+forwarding shim cannot, and **66 of the 222 sites are spelled that way**. A
 context resolved from an inline literal or a local `const` can be tested for
 `isSystem`; one arriving from a helper call cannot.
 
@@ -126,8 +126,8 @@ now **0**: nothing on this surface threads a context that provably lacks the fla
 
 **"No tenant context" counted sites it had not read.** An options argument the
 walker could not parse was folded into the same bucket as one it had read and
-found empty. That published **84 sites "carrying no tenant context at all"**
-when 17 said so and 67 were simply unread — an over-claim in the *alarming*
+found empty. That published **83 sites "carrying no tenant context at all"**
+when 17 said so and 66 were simply unread — an over-claim in the *alarming*
 direction, on the very figure this page tells other cards to cite. `carries` is
 now three-valued, and an unreadable argument can never contribute to the
 provable count.
@@ -147,10 +147,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
 
 | carried figure | where it survives | this census |
 | :--- | :--- | ---: |
-| 175 write call sites | quoted in the merged changeset | **223** |
+| 175 write call sites | quoted in the merged changeset | **222** |
 | 24 carrying no tenant context | quoted in the merged changeset | **9** provable and tenancy-enabled; **32** more whose options argument is unreadable |
-| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **149 of 223** decidable, **74** undecidable |
-| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 105 decidably elevated, 0 decidably not, 101 undecidable |
+| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **149 of 222** decidable, **73** undecidable |
+| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 105 decidably elevated, 0 decidably not, 100 undecidable |
 | 141 and 132, two independent re-derivations | the card that filed this work | — |
 
 **The differences are not reconciled, and deliberately so.** The old census's
@@ -167,11 +167,11 @@ would report a smaller number and would not say so.
 
 The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
 figure has no surviving corroboration anywhere in the tree.** This census reads
-105 of 223 (47%) as decidably elevated, with 101 more whose elevation is a
+105 of 222 (47%) as decidably elevated, with 100 more whose elevation is a
 run-time fact — so the claim is neither confirmed nor refuted, and the honest
 answer is that a static reading cannot settle it.
 
-⇒ **Cite `9 / 223`, and say what it is**: the sites whose options argument was
+⇒ **Cite `9 / 222`, and say what it is**: the sites whose options argument was
 READ and holds no tenant context, against a decidably tenancy-enabled object.
 That is the control's provable yield surface. ⛔ Do not cite it as "the sites
 without tenant context" — **32 further sites** have an options argument this
@@ -183,31 +183,31 @@ cannot read, and they are neither in nor out.
 
 | what | count |
 | :--- | ---: |
-| write call sites on the application surface | **223** |
+| write call sites on the application surface | **222** |
 | …whose object name is statically decidable | 149 |
-| …whose object name is chosen at run time | 74 |
+| …whose object name is chosen at run time | 73 |
 | …against an object with tenancy ENABLED | 149 |
 | …against an object that declares tenancy off | 0 |
 | threading a tenant context | 139 |
 | PROVABLY carrying none (options read, no context key) | **17** |
 | …of those, against a decidably tenancy-enabled object | **9** |
-| options argument UNREADABLE — may or may not carry one | 67 |
+| options argument UNREADABLE — may or may not carry one | 66 |
 | …of those, against a decidably tenancy-enabled object | 32 |
 | threading a decidably ELEVATED (`isSystem`) context | 105 |
 | threading a context that is decidably NOT elevated | 0 |
-| threading a context whose elevation is a run-time fact | 101 |
+| threading a context whose elevation is a run-time fact | 100 |
 
 | how the instrument reached the site | count |
 | :--- | ---: |
-| receiver carried a readable engine type | 178 |
+| receiver carried a readable engine type | 177 |
 | receiver erased, placed by the object NAME | 19 |
 | receiver erased, placed by an `object: string` PARAMETER | 15 |
 | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 11 |
 
 | object name spelled inline | 109 |
 | object name spelled through a `const` | 40 |
 | object name is an `object: string` parameter | 19 |
-| object name is some other run-time expression | 55 |
+| object name is some other run-time expression | 54 |
 
 The corpus walked is every tracked non-test source under `packages/services/`
 and `packages/plugins/`; calls to a same-named method on something that is not
@@ -224,12 +224,12 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-09-10 at `638d2b544`.
+Measured on 2026-09-11 at `f368fd0aa`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 562 |
-| engine-shaped types recognised | 58 |
+| tracked non-test sources scanned | 563 |
+| engine-shaped types recognised | 59 |
 | declared objects in the registry | 300 |
 | same-named calls subtracted as non-engine | 137 |
 

docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

@@ -29,19 +29,19 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.
 
 | Measure | Value |
 |---|---:|
-| Write call sites | 223 |
+| Write call sites | 222 |
 | Object name statically decidable | 149 |
-| Object name chosen at run time | 74 |
+| Object name chosen at run time | 73 |
 | Against a tenancy-enabled object | 149 |
 | Against an object declaring tenancy off | 0 |
 | Threading a tenant context | 139 |
 | Provably carrying none | 17 |
 | …and decidably tenancy-enabled | 9 |
-| Options argument unreadable | 67 |
+| Options argument unreadable | 66 |
 | …and decidably tenancy-enabled | 32 |
 | Threading a decidably elevated context | 105 |
 | Threading a decidably non-elevated context | 0 |
-| Threading a context of undecidable elevation | 101 |
+| Threading a context of undecidable elevation | 100 |
 
 ## Corpus scale — present and dated, ⛔ NOT enforced
 
@@ -52,12 +52,12 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-09-10 at `638d2b544`.
+Measured on 2026-09-11 at `f368fd0aa`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 562 |
-| engine-shaped types recognised | 58 |
+| tracked non-test sources scanned | 563 |
+| engine-shaped types recognised | 59 |
 | declared objects in the registry | 300 |
 | same-named calls subtracted as non-engine | 137 |
 
@@ -181,7 +181,7 @@ Measured on 2026-09-10 at `638d2b544`.
 | `packages/services/service-messaging/src/sql-http-outbox.ts` | `insert` | `this.objectName` | undecidable | options unreadable | 1 |
 | `packages/services/service-messaging/src/sql-http-outbox.ts` | `update` | `this.objectName` | undecidable | options unreadable | 4 |
 | `packages/services/service-messaging/src/sql-outbox.ts` | `insert` | `this.objectName` | undecidable | options unreadable | 1 |
-| `packages/services/service-messaging/src/sql-outbox.ts` | `update` | `this.objectName` | undecidable | options unreadable | 5 |
+| `packages/services/service-messaging/src/sql-outbox.ts` | `update` | `this.objectName` | undecidable | options unreadable | 4 |
 | `packages/services/service-queue/src/db-queue-adapter.ts` | `delete` | `sys_job_queue` | enabled | context, elevation undecidable | 2 |
 | `packages/services/service-queue/src/db-queue-adapter.ts` | `insert` | `sys_job_queue` | enabled | context, elevation undecidable | 1 |
 | `packages/services/service-queue/src/db-queue-adapter.ts` | `update` | `sys_job_queue` | enabled | context, elevation undecidable | 6 |

packages/services/service-messaging/src/delivery-claim-tenant-audit.integration.test.ts

@@ -219,4 +219,30 @@ describe('sys_notification_delivery — the dispatcher claim path is a classifie
 
         await controlUnscopedUpdateMany(DELIVERY_OBJECT);
     });
+
+    it("reap() recovers a crashed node's claims in every organization, without a finding", async () => {
+        const outbox = new SqlNotificationOutbox(engine as any, { partitionCount: 1 });
+        const idA = await outbox.enqueue({
+            notificationId: 'n_a', recipientId: 'u_a', channel: 'inbox', organizationId: 'org_a', payload: {},
+        } as any);
+        const idB = await outbox.enqueue({
+            notificationId: 'n_b', recipientId: 'u_b', channel: 'inbox', organizationId: 'org_b', payload: {},
+        } as any);
+        // A node claims both rows and dies: its claim is stamped ten minutes ago.
+        await outbox.claim({ nodeId: 'dead_node', limit: 10, claimTtlMs: 60_000, now: Date.now() - 10 * 60_000 });
+
+        // [#17610] The dispatcher's once-per-tick reap, on its own — a third
+        // predicate write on the claim path, classified by the same warrant.
+        await outbox.reap({ claimTtlMs: 60_000 });
+
+        expect(auditedUpdateMany(DELIVERY_OBJECT)).toBe(false);
+        // Both organizations' abandoned rows are back in the queue with the claim
+        // credential cleared — a per-organization reap would have stranded one.
+        const rows = await outbox.list();
+        expect(rows.map((r) => `${r.id}:${r.organizationId}:${r.status}:${r.claimedBy ?? '-'}`).sort()).toEqual(
+            [`${idA}:org_a:pending:-`, `${idB}:org_b:pending:-`].sort(),
+        );
+
+        await controlUnscopedUpdateMany(DELIVERY_OBJECT);
+    });
 });