feat(objectql,cli): os migrate 新增 summary count/sum 存量 NULL 回填迁移 (#6063)

.changeset/summary-null-backfill-migration.md

@@ -0,0 +1,51 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/cli": patch
+---
+
+feat(objectql,cli): `os migrate summary-nulls` backfills roll-up count/sum columns left NULL by pre-seed inserts (#6063)
+
+#5749 / PR #6013 fixed the **producer**: a parent row created from that release
+on has its `count` / `sum` roll-up columns seeded to the empty-set value at
+insert, so `filter ["task_count", "=", 0]`, sorting, `GROUP BY` and formulas
+over the column stop silently dropping parents that never had a child.
+
+Being a create-time fix, it reaches **new rows only**. A database upgraded **in
+place** still holds parents stored before the upgrade, and the recompute that
+would otherwise correct them runs only when one of their **children** is
+written — so those rows keep their `NULL` indefinitely and keep disappearing
+from the same queries. A freshly seeded deployment is correct; an upgraded one
+is not. This release ships the other half: a one-off, explicit data migration.
+
+```bash
+os migrate summary-nulls                    # dry run: full report, writes nothing
+os migrate summary-nulls --apply            # recompute and write (prompts)
+os migrate summary-nulls --apply --yes --json   # CI / scripts
+os migrate summary-nulls --object project   # restrict to one object (repeatable)
+```
+
+**Every NULL row is recomputed, never blanket-set to 0.** A pre-upgrade parent
+that *does* have children is `NULL` too — nothing ever recomputed it — and its
+correct value is the real aggregate. `UPDATE ... SET col = 0 WHERE col IS NULL`
+would replace a visibly-missing value with a confidently-wrong one, which the
+next child write would then silently change back. The run computes each value
+through the same code path the engine's own child-write recompute uses
+(`aggregateSummaryValue`), over the descriptors the engine itself maintains, so
+a backfilled column and a recomputed one can never mean different things.
+
+**`min` / `max` / `avg` are never touched.** They are undefined on an empty set
+— which is why the insert-time seed leaves them `null` — so a stored `null`
+there is the correct reading of "no child rows", not a defect. The report names
+them as deliberately skipped rather than omitting them silently.
+
+Other properties: dry run by default and a dry run writes nothing at all;
+idempotent, so re-running is safe and a clean report is the operator's own
+verification; driver-agnostic (it reads values and tests them in JS rather than
+pushing a null predicate down, since null-predicate compilation is precisely
+where drivers diverge); one row's failure is recorded and the run carries on.
+It records no deployment flag — unlike its `os migrate` siblings, nothing is
+gated on it having run.
+
+Never running it is safe in the sense that nothing breaks *further*: the
+affected rows simply stay missing from `= 0` filters until a child of theirs is
+written.

content/docs/deployment/cli.mdx

@@ -659,6 +659,7 @@ where the data lives.
 |---------|-------------|
 | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
 | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean |
+| `os migrate summary-nulls` | Backfill roll-up `count` / `sum` columns still stored as `NULL` on parent rows created before the insert-time seed. Repairs values; no flag, nothing depends on it having run |
 | `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run |
 
 ```bash
@@ -755,6 +756,43 @@ Same writing rules as its sibling: a dry run writes **nothing**, `--apply` is
 the only writing mode, a later failing run clears the verified state, and a
 running server reads the flag once — **restart it** after a successful apply.
 
+#### `os migrate summary-nulls`
+
+A roll-up `summary` field of function `count` or `sum` is **0** over an empty
+child collection — zero children is zero, not "unknown" — and since #5749 a
+parent row is created holding that value. Rows created *before* that are the
+exception: nothing seeded them, and the recompute that maintains a roll-up runs
+only when one of the parent's **children** is written, so a parent that has
+never had a child keeps its `NULL` indefinitely. `filter ["task_count", "=", 0]`
+then silently omits it, and so do sorting, `GROUP BY` and any formula reading
+the column (null propagation).
+
+```bash
+os migrate summary-nulls                    # Dry run: full report, writes nothing
+os migrate summary-nulls --apply            # Recompute and write (prompts)
+os migrate summary-nulls --apply --yes --json   # CI / scripts
+os migrate summary-nulls --object project       # Restrict to one object (repeatable)
+```
+
+**Each affected row is recomputed, not set to 0.** A pre-upgrade parent that
+*does* have children is `NULL` too, and its correct value is the aggregate over
+them — writing 0 there would replace a missing value with a wrong one, and the
+next child write would change it back. The report separates the two: `N NULL
+row(s), M with real child data`.
+
+`min` / `max` / `avg` are **never touched**. They are undefined on an empty set,
+so a `null` there is the correct reading of "no child rows"; the report lists
+them as deliberately skipped.
+
+Idempotent — every write turns a `NULL` into a number, so a second run finds
+nothing and writes nothing. Re-running until the report says zero *is* the
+verification, which is why this command records no flag: it repairs values and
+changes no behaviour, so there is no posture for a flag to attest.
+
+A deployment whose database was seeded fresh on this version has nothing to do
+here: its parents were created with the value already in place, and the run
+reports zero.
+
 #### A database created by this version needs no migration
 
 A deployment whose database the platform **creates from empty** records these

packages/cli/src/commands/migrate/summary-nulls.ts

@@ -0,0 +1,248 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+import { Command, Flags } from '@oclif/core';
+import chalk from 'chalk';
+import { createInterface } from 'node:readline';
+import {
+  printHeader,
+  printSuccess,
+  printWarning,
+  printError,
+  printInfo,
+  printStep,
+  createTimer,
+  emitJson,
+  isExitSignal,
+} from '../../utils/format.js';
+import { bootSchemaStack } from '../../utils/schema-migrate.js';
+import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js';
+import { describeOccupancy } from '../../utils/sqlite-occupancy.js';
+import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js';
+// Type-only, so the heavy engine package is still loaded lazily below: this is
+// the surface the migration actually needs, which is wider than the `objectql`
+// slot contract by exactly one member (`getOwnedSummaryDescriptors`). Naming it
+// here keeps the call site checked instead of erasing the lookup to `any`
+// (#4168/#4251).
+import type { SummaryBackfillEngine } from '@objectstack/objectql';
+
+async function confirm(question: string): Promise<boolean> {
+  if (!process.stdin.isTTY) return false; // non-interactive → require --yes
+  const rl = createInterface({ input: process.stdin, output: process.stdout });
+  try {
+    const answer: string = await new Promise((resolve) => rl.question(question, resolve));
+    return /^y(es)?$/i.test(answer.trim());
+  } finally {
+    rl.close();
+  }
+}
+
+/**
+ * `os migrate summary-nulls` — the one-off backfill of roll-up `count`/`sum`
+ * columns left `NULL` by inserts predating PR #6013 (#6063, the second half of
+ * #5749).
+ *
+ * PR #6013 fixed the producer: a parent created from that release on starts its
+ * `count`/`sum` roll-ups at 0. It cannot reach rows that already exist, and the
+ * recompute only ever visits a parent when one of its CHILDREN is written — so
+ * a database upgraded IN PLACE keeps pre-upgrade parents at `NULL`, and every
+ * `= 0` filter, sort, GROUP BY and formula over the column silently drops them.
+ * A freshly seeded database has no such rows; this command is for the other
+ * kind, and is needed exactly once per deployment.
+ *
+ * Dry run by default (writes nothing at all), `--apply` to backfill. Each
+ * `NULL` row is RECOMPUTED — a pre-upgrade parent that has children is `NULL`
+ * too and its correct value is the real aggregate, so writing 0 everywhere
+ * would swap a missing value for a wrong one. Idempotent: a second run finds
+ * nothing left to do.
+ *
+ * `min`/`max`/`avg` are never touched — undefined on an empty set, so a `null`
+ * there is the correct reading of "no child rows", not a defect.
+ *
+ * ## No deployment flag, deliberately
+ *
+ * Its siblings (`files-to-references`, `value-shapes`) record a `sys_migration`
+ * flag because that flag is what later OPENS irreversible behaviour on the
+ * deployment. Nothing is gated on this run: it repairs values and changes no
+ * posture, and its own idempotence is the verification (re-run; a clean report
+ * is the evidence). A flag here would be a fact nothing reads.
+ */
+export default class MigrateSummaryNulls extends Command {
+  static override description =
+    'Backfill roll-up count/sum summary columns still stored as NULL on parent rows created before the ' +
+    'insert-time seed (#5749). Dry-run by default; --apply recomputes and writes each affected row.';
+
+  static override examples = [
+    '$ os migrate summary-nulls',
+    '$ os migrate summary-nulls --apply',
+    '$ os migrate summary-nulls --apply --yes --json',
+    '$ os migrate summary-nulls --object project',
+  ];
+
+  static override flags = {
+    'database-url': Flags.string({
+      description: 'Database URL to migrate (defaults to $OS_DATABASE_URL / the project DB)',
+      env: 'OS_DATABASE_URL',
+    }),
+    apply: Flags.boolean({
+      description: 'Write the recomputed values (default is a read-only dry run)',
+      default: false,
+    }),
+    yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }),
+    force: Flags.boolean({
+      description: 'Apply even when another process is using the database (SQLite occupancy check)',
+      default: false,
+    }),
+    object: Flags.string({
+      description: 'Restrict to this object (repeatable; default: every object owning a count/sum roll-up)',
+      multiple: true,
+    }),
+    'max-records': Flags.integer({
+      description: 'Safety bound on parent rows read per object — exceeding it truncates the walk',
+    }),
+    json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }),
+  };
+
+  async run(): Promise<void> {
+    const { flags } = await this.parse(MigrateSummaryNulls);
+    const timer = createTimer();
+    const apply = flags.apply;
+
+    if (!flags.json) printHeader('Migrate · summary-nulls');
+
+    // Occupancy gate, like `files-to-references`: an apply run rewrites ROWS,
+    // so a second writer on the same SQLite file is a real hazard. Probed
+    // before boot (afterwards our own pool is what the probe finds) and before
+    // the prompt, so an operator is never asked to confirm a run we refuse.
+    const occupancy = await probeMigrationTarget(flags['database-url']);
+    if (occupancy.status === 'busy' && apply && !flags.force) {
+      if (flags.json) {
+        await emitJson({
+          error: 'database_busy',
+          database: occupancy.filename,
+          signal: occupancy.signal,
+          detail: occupancy.detail,
+          hint: OCCUPANCY_HINT,
+        }, 0, { compact: true });
+        this.exit(1);
+        return;
+      }
+      printError(describeOccupancy(occupancy));
+      printWarning(OCCUPANCY_HINT);
+      this.exit(1);
+      return;
+    }
+    if (occupancy.status === 'busy' && !flags.json) {
+      printWarning(apply
+        ? `--force: ${describeOccupancy(occupancy)} Backfilling anyway — the live process may write rows mid-walk.`
+        : `${describeOccupancy(occupancy)} The dry run below writes nothing, but its counts may shift while that process is running.`);
+    }
+    if (occupancy.status === 'unknown' && !flags.json) {
+      printWarning(`Could not check whether the database is in use — ${occupancy.detail}`);
+    }
+
+    if (apply && !flags.yes) {
+      if (flags.json || !process.stdin.isTTY) {
+        if (flags.json) {
+          await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true });
+          this.exit(1);
+          return;
+        }
+        printWarning('Apply mode rewrites record data. Re-run with --yes to confirm, or run without --apply to preview.');
+        this.exit(1);
+        return;
+      }
+      const ok = await confirm(
+        chalk.bold('\nRecompute and write every NULL count/sum roll-up value on this database? [y/N] '),
+      );
+      if (!ok) {
+        printInfo('Aborted — no changes made.');
+        return;
+      }
+    }
+
+    if (!flags.json) {
+      printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (dry run)…');
+    }
+
+    let stack;
+    try {
+      stack = await bootSchemaStack({
+        databaseUrl: flags['database-url'],
+        extraPlugins: await buildDataMigrationPlugins(),
+      });
+    } catch (error: any) {
+      if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); }
+      printError(error.message || String(error));
+      this.exit(1);
+      return;
+    }
+
+    try {
+      const engine: SummaryBackfillEngine = stack.kernel.getService('objectql');
+      if (typeof engine?.getOwnedSummaryDescriptors !== 'function') {
+        throw new Error('No ObjectQL engine on this stack — cannot read the roll-up index.');
+      }
+      // An empty walk is indistinguishable from a clean one, and "clean" is the
+      // answer an operator will act on — so refuse to run when no app metadata
+      // is loaded (missing artifact / wrong directory) rather than report a
+      // database the walk never looked at.
+      const loadedObjects: string[] =
+        typeof engine.getConfigs === 'function' ? Object.keys(engine.getConfigs()) : [];
+      if (!loadedObjects.some((name) => !name.startsWith('sys_'))) {
+        throw new Error(
+          'No app objects are loaded, so the walk would examine nothing. ' +
+            'Run "os build" in your project root first (the migration reads dist/objectstack.json), then re-run.',
+        );
+      }
+
+      const { backfillSummaryNulls, formatSummaryBackfillReport } = await import('@objectstack/objectql');
+
+      // In JSON mode keep stdout parseable — route warnings to stderr.
+      const logger = flags.json
+        ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) }
+        : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) };
+
+      const report = await backfillSummaryNulls(engine, logger, {
+        apply,
+        objects: flags.object,
+        maxRecordsPerObject: flags['max-records'],
+      });
+
+      if (flags.json) {
+        await emitJson({ database: stack.dbLabel, apply, report, duration: timer.elapsed() });
+        if (report.failures.length > 0) this.exit(1);
+        return;
+      }
+
+      printInfo(`Database: ${chalk.white(stack.dbLabel)}`);
+      console.log('');
+      console.log(formatSummaryBackfillReport(report).join('\n'));
+      console.log('');
+
+      if (report.failures.length > 0) {
+        printError(`${report.failures.length} row(s) could not be recomputed — re-run to finish them.`);
+      } else if (apply && report.filled > 0) {
+        printSuccess(
+          `Backfilled ${report.filled} roll-up value(s) across ${report.fields.length} column(s). ` +
+            'Re-run any time — it only ever revisits rows still holding NULL.',
+        );
+      } else if (apply) {
+        printSuccess('Nothing to backfill — every count/sum roll-up already holds a value.');
+      } else if (report.nullRows > 0) {
+        printInfo(`Dry run only — ${report.nullRows} value(s) would be recomputed. Re-run with --apply.`);
+      } else {
+        printSuccess('Nothing to backfill — every count/sum roll-up already holds a value.');
+      }
+      console.log(chalk.dim(`  ${timer.display()}`));
+      console.log('');
+      if (report.failures.length > 0) this.exit(1);
+    } catch (error: any) {
+      if (isExitSignal(error)) throw error;
+      if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); }
+      printError(error.message || String(error));
+      this.exit(1);
+    } finally {
+      await stack.shutdown();
+    }
+  }
+}