Skip to content

feat(objectql,cli): os migrate 新增 summary count/sum 存量 NULL 回填迁移 (#6063) - #6158

Merged
baozhoutao merged 4 commits into
mainfrom
claude/issue-6063-summary-null-backfill
Aug 7, 2026
Merged

feat(objectql,cli): os migrate 新增 summary count/sum 存量 NULL 回填迁移 (#6063)#6158
baozhoutao merged 4 commits into
mainfrom
claude/issue-6063-summary-null-backfill

Conversation

@baozhoutao

Copy link
Copy Markdown
Contributor

Fixes #6063

前提复核(P1-P3,实读 origin/main)

P1 —— #6013 的口径确如裁决描述。成立。

packages/objectql/src/engine.ts(origin/main,合并前行号):

  • 829: function summaryEmptySetValue(fn) { return fn === 'count' || fn === 'sum' ? 0 : null; } —— count/sum 的空集值是 0,min/max/avg 是 null,原文自陈是「the ONE place that list is written down」;
  • 4350: private initializeSummaryFields(...) 在 insert 路径上按该表播种,文档串 4347 行原文:「Existing rows are untouched: this is create-time only, so parents already stored with null stay null until a child write recomputes them.」
  • 4410: if (value == null) value = summaryEmptySetValue(desc.fn); —— 重算落到同一张表。
  • 测试侧:summary-rollup.test.ts:286 起的 roll-up summary seeding on the PARENT insert (#5749) 整块,含 375leaves avg/max null — undefined on an empty set

⇒ 裁决描述的「insert 起 count/sum 为 0、min/max/avg 保持 null」逐字成立。

P2 —— 存量 NULL 父行在子写入前不会被任何既有路径补齐。成立。

recomputeSummaries 的父行集合只来自子记录:idsrecs / prevsdesc.fkField 构成(engine.ts 4372 起),没有第三个来源;initializeSummaryFields 是 create-time only(上引 4347)。全仓再无别处写这些列。行为侧也钉住了:本 PR 新测 dry run reports the same rows and writes nothing 里,直写驱动 store 造出的存量父行在回填前读回仍是 null

P3 —— 迁移家族结构如裁决引用。成立。

  • 命令壳:packages/cli/src/commands/migrate/{value-shapes,files-to-references}.ts;
  • 真迁移在归属包:scanValueShapes@objectstack/objectql,runFilesToReferencesMigration@objectstack/service-storage;
  • 显式跑 + 产出证据:dry-run 默认、--apply 才写、报告函数 format*Report 由命令壳打印。

三条前提全部为真,未改道。

裁决引用(issue #6063,2026-08-07 02:07Z)

  1. 显式 os migrate,不开机自动跑 —— 落 os migrate summary-nulls,命令壳在 packages/cli/src/commands/migrate/summary-nulls.ts(薄),真迁移在 objectql;
  2. 逐行重算聚合,便宜路线默认禁用 —— 本 PR 采用便宜路线,亦尝试证伪反例:恰恰相反,新测第 1 例正是「NULL 但有子记录」的活体,便宜路线在它上面翻红(见下反向验证);
  3. 引擎层驱动无关循环 —— 走引擎 find / aggregate / update,零逐驱动 SQL,且不下推 IS NULL 谓词(理由见下)。

处置

新增

  • packages/objectql/src/summary-backfill.ts —— backfillSummaryNulls(engine, logger, options) + formatSummaryBackfillReport + summaryBackfillComplete。遍历「拥有 count/sum 汇总列」的对象,keyset 走全部父行(只取 id + 汇总列),对读回值为 null 的行逐行重算并写回。dry-run 默认;幂等;单行失败记录后继续。
  • packages/objectql/src/summary-aggregate.ts —— 从 engine.ts 原样搬出的 roll-up 内核:SummaryDescriptorsummaryEmptySetValue、单描述符聚合 aggregateSummaryValue,外加派生谓词 summaryNullIsBackfillable。descriptor 新增 childObject 字段(父侧索引此前拿不到子对象名,因为子侧索引就是按它做 key)。
  • packages/cli/src/commands/migrate/summary-nulls.ts —— 照 files-to-references 的壳:occupancy gate(它改行)、--apply / --yes / --force / --object / --max-records / --json、空元数据拒跑。
  • packages/objectql/src/summary-backfill.test.ts —— 10 例。
  • changeset(objectql + cli,patch)、content/docs/deployment/cli.mdx 的数据迁移小节。

为什么不写 UPDATE ... SET col = 0 WHERE col IS NULL

它不只是粗,它是错的:升级前就有子记录的父行同样是 NULL(从没有人重算过它),正确值是真实聚合而不是 0。写 0 等于把「可见地缺失」换成「自信地错误」,而下一次子写入又会把它悄悄改回去 —— 用户什么都没动,列却变了。

为什么不下推 IS NULL

null 谓词的编译正是各驱动分歧最大的地方(sql-driver-null-operators.test.tssql-driver-out-of-contract-filter-input.test.ts 的分歧表)。一条覆盖范围取决于「碰巧跑在哪个驱动上」的迁移,等于会静默漏行的迁移。读回值在 JS 里判 == null,是唯一在所有驱动上含义相同的写法;代价见「性能口径」。

不记 migration flag

两个兄弟迁移记 flag,是因为那面 flag 是后续不可逆行为的开关。本迁移只修值、不改任何 posture,幂等重跑本身就是验证(报告归零即证),记一面没人读的 flag 反而是噪音。已写进模块与命令的文档串。

测试与反向验证

新测(10 例,均绿)

✓ gives a NULL parent WITH children its real aggregate — not 0
✓ gives a NULL parent with NO children the empty-set value 0
✓ leaves min/max/avg NULL exactly as they are, and reports them as out of scope
✓ is idempotent — the second run finds nothing and writes nothing
✓ is a no-op on a database whose rows were all created with the seed (a fresh install)
✓ dry run reports the same rows and writes nothing
✓ never overwrites a value already stored — including a deliberate 0
✓ writes the SAME value the engine's own child-write recompute would
✓ restricts to the objects it is given
✓ records a row it cannot write and carries on with the rest

存量状态是直写驱动 store 造出来的 —— 那正是原地升级库的样子(父行早于播种存在,且从没有子写入访问过它)。走引擎 insert 造反而会被 #6013 播成 0,就没得测了。

反向验证(方向先写死,再运行)

肢 A —— 把回填临时改成便宜路线 computed = 0

  • 预测:用例 1「NULL 且子记录 → 真实聚合」翻红。
  • 实测:翻红,与预测一致,并且多红一例(方向相同,幅度更大):
× gives a NULL parent WITH children its real aggregate — not 0
  → expected +0 to be 2
× writes the SAME value the engine's own child-write recompute would
  → expected 42 to be +0
Tests  2 failed | 8 passed (10)

第二例是意外收获,也是本单最该被看见的伤害形态:便宜路线写了 0 之后,下一次子写入的重算把列改回 42 —— 用户没动任何东西,列自己变了。同源同算的钉子就是钉这个。已恢复原实现,上述 10 例复绿。

肢 B —— 注册接线。

  • 预测:迁移未注册时 os migrate 列表不含它;注册后含。
  • 实测,两个方向都跑了(oclif 按 dist/commands/**/*.js glob 发现,所以「未注册」= 把编译产物移走):
# 移走 dist/commands/migrate/summary-nulls.js
$ node bin/run.js migrate --help | grep -c summary-nulls
0
$ node bin/run.js migrate summary-nulls --help
 ›   Error: Command migrate:summary-nulls not found.

# 放回
$ node bin/run.js migrate --help | grep -c summary-nulls
1

注册后的列表项(真实输出节选):

  migrate summary-nulls        Backfill roll-up count/sum summary columns still
                               stored as NULL on parent rows created before the
                               insert-time seed (#5749). Dry-run by default;
                               --apply recomputes and writes each affected row.

命令与真实输出

合并 origin/main(⛔ 未 rebase)后重跑,origin/main 带进了 #5979@objectstack/metadata/errors 新依赖边,worktree 补跑了一次 pnpm install(与本 PR 无关,记录在此以免下一位读者误判):

$ pnpm --filter @objectstack/objectql test -- --maxWorkers=2
 Test Files  133 passed (133)
      Tests  2190 passed (2190)

$ pnpm --filter @objectstack/objectql typecheck
> tsc --noEmit                       # 无输出即通过

$ pnpm --filter @objectstack/cli test -- --maxWorkers=2
 Test Files  88 passed (88)
      Tests  880 passed (880)

$ pnpm --filter @objectstack/cli typecheck
> tsc --noEmit

$ node scripts/check-engine-double-contract.mjs      # exit=0
check-engine-double-contract: OK — 73 pinned, 133 in the DEBT ledger, 2 exempt.

$ node scripts/check-nul-bytes.mjs
check-nul-bytes: OK (scanned 5882 tracked text file(s); ... no raw ASCII control bytes).

$ pnpm check:query-options-erasure
✓ query-options-erasure ratchet holds: 83 unswept non-test site(s) in 19 file(s), none new.

$ pnpm check:doc-authoring
✓ doc authoring guard: 363 files clean — no bare metadata literals.

$ pnpm check:empty-changeset
✓ No empty-frontmatter changeset introduced by this diff (1 declaring changeset(s) added).

$ pnpm check:durability-log-level
✓ durability-degradation log levels: 24 durability-critical catch seam(s), all loud ...

check:query-options-erasure 中途红过一次并已修:搬运时把 as any 一起带了过来,该闸门拒绝新文件进 baseline(grandfather 列表只缩不增)。修法是把它去掉而不是加白名单 —— 新接口 SummaryAggregateEngine.aggregate 直接收该 query,本来就不需要断言。baseline 最终只记录 engine.ts 自己的计数 13 → 12。

必答项

1. #6013 口径一致性:回填值 / insert 初值 / 重算值三者同源同算?

是,且是结构性同源,不是「今天恰好相等」。 三个调用点读同一份代码:

  • insert 初值:ObjectQL.initializeSummaryFieldssummaryEmptySetValue(desc.fn);
  • 子写入重算:ObjectQL.recomputeSummariesaggregateSummaryValue(this, desc, parentId, execCtx),该函数内部空集回退同样是 summaryEmptySetValue(desc.fn);
  • 本回填:backfillSummaryNulls同一个 aggregateSummaryValue(engine, desc, ...)

descriptor 也同源:回填不自己解析 summary 元数据,而是调 engine.getOwnedSummaryDescriptors(object)引擎自己的索引(同一份 FK 解析、同一份 filter、同一条 objectRevision 失效规则)。范围收窄谓词 summaryNullIsBackfillable(fn)summaryEmptySetValue(fn) !== null 派生,不另列函数名单,所以两者不可能各说各话。

行为侧还额外钉了一道:测试 writes the SAME value the engine's own child-write recompute would —— 回填后再触发一次真实子写入重算,断言列不动。肢 A 证明这枚钉子会红。

2. #5979 / #5351 / #6014 交叠

未触其面。 逐条:

engine.ts 的 hunk 逐个点名(共 4 处,全部落在 roll-up 机制内部,与上述两单的落点 seedAutonumber / buildDriverOptions 不相邻):

  1. import 段:新增 ./summary-aggregate.js 的三个符号;
  2. SummaryDescriptor + summaryEmptySetValue 定义处 → 换成一条指向新文件的说明注释(纯搬运,零语义改动);
  3. buildSummaryIndex 的 descriptor 字面量:补 childObject(值就是本来当索引 key 用的那个变量);
  4. getOwnedSummaryDescriptors 去掉 private;recomputeSummaries 内联的聚合块 → 调 aggregateSummaryValue

共享 index packages/objectql/src/index.ts 也碰了(导出新迁移面),追加式,不改既有行。

之所以不是「新文件里再写一遍聚合」:那正是 scan-value-shapes.ts 头注释「Why the predicate is imported」写下的教训 —— 第二份实现只要差一个从句,迁移写进去的值就和引擎下一次重算的值不一致。必答项 1 要求「必须同源」,搬运是唯一能给出代码证据的做法。

3. 多租户形态

value-shapes / files-to-references 完全一致,不新造机制:

  • 读写都带 { isSystem: true }不带 tenantIdbuildDriverOptions(engine.ts)只在 execCtx.tenantId !== undefined 时才下发租户作用域,所以本迁移在共享库多 org 形态下一次覆盖库内所有 org 的行 —— 这正是部署级回填需要的;
  • 按 org 分库 / 分表 形态下,与两个兄弟迁移一样:每个库跑一次,用 --database-url(或 OS_DATABASE_URL)指定。命令的 occupancy gate、确认提示与报告都是按「一次一个目标库」设计的;
  • 迁移只跑「已加载元数据里拥有 count/sum 汇总列的对象」,无对象加载时直接拒跑(空扫和干净扫不可区分)。

4. 性能口径

  • 聚合次数 = O(存量 NULL 的 父行 × count/sum 汇总列 对数),与裁决口径一致。已回填的行第二次跑不再产生任何聚合(幂等),所以量级只随「真实缺口」走,不随库大小走;
  • 读取 = O(父行总数),分页 keyset(pageSize 500,只取 id + 汇总列)。这是为了不下推 IS NULL 而付的代价,与 scanValueShapes(500/页)、backfillFileReferences(200/页)同形,先例照抄;
  • 批处理策略:先例没有批量聚合的形态,本迁移也不引入 —— 单行 aggregate + 单行 update,外加与引擎重算同款的 withTransientRetry(bug(objectql): 汇总重算失败仅 warn——子记录批量插入报成功,父 summary 静默过期且无重试 #3147)。这样一行失败只损失一行,而且重跑只会回访仍是 NULL 的行;
  • 兜底:--max-records 可为每个对象设上限,超限报告 truncated,不假装跑完;--object 可只跑已知受影响的对象;
  • 量级估算:10 万父行 / 2 个汇总列、其中 1% 为 NULL 的库,一次运行约 20 万行读(400 页)+ 2000 次聚合 + 2000 次 update;第二次运行退化为纯 400 页读、零写。真正大的库建议先 dry-run 看 N NULL row(s) 再决定窗口。

顺带

无越界发现,未新开 issue。⛔ 未碰 content/docs/releases/(known-issue 文案由发布流程从 issue 取材)。


Generated by Claude Code

claude added 3 commits August 7, 2026 03:28
…count/sum roll-ups (#6063)

PR #6013 (#5749) seeds a roll-up's empty-set value at parent INSERT, which
reaches new rows only: a database upgraded in place keeps pre-upgrade parents
at NULL, because the recompute that would fix them runs only when one of their
children is written. Those rows keep vanishing from `= 0` filters, sorts,
GROUP BY and formulas. This adds the one-off, explicit data migration for them.

- `backfillSummaryNulls` (packages/objectql/src/summary-backfill.ts): walk each
  object owning a count/sum roll-up, and recompute every row whose column is
  stored NULL. A pre-upgrade parent WITH children is NULL too and its correct
  value is the real aggregate, so `SET col = 0 WHERE col IS NULL` is wrong, not
  merely coarse. Dry run by default; idempotent; driver-agnostic (values are
  read and tested in JS, no null predicate pushed down); one row's failure is
  recorded and the run continues.
- min/max/avg are never touched: undefined on an empty set, so a stored null
  there is the correct reading of "no child rows". The report names them as
  deliberately skipped.
- `summary-aggregate.ts`: SummaryDescriptor, summaryEmptySetValue and the
  single-descriptor aggregate lifted out of engine.ts unchanged, so the seed,
  the recompute and the backfill share ONE computation instead of three that
  agree until one is edited. The descriptor gains `childObject` so the
  parent-side index is usable on its own.
- `os migrate summary-nulls`: thin oclif shell over the migration, following
  the files-to-references precedent (occupancy gate, --apply/--yes, --object,
  --max-records, --json). No deployment flag — nothing is gated on this run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We
…ing it (#6063)

`check:query-options-erasure` rejects a NEW file in its baseline — the
grandfather list only shrinks. The `as any` came along with the code lifted out
of engine.ts and is not needed there: `SummaryAggregateEngine.aggregate` takes
the query directly, so the cast is dropped and the baseline records only
engine.ts's own count falling 13 -> 12.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We
@vercel

vercel Bot commented Aug 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Aug 7, 2026 4:07am

Request Review

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/cli, @objectstack/objectql.

31 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/ai/skills-reference.mdx (via packages/cli)
  • content/docs/api/client-sdk.mdx (via @objectstack/cli)
  • content/docs/api/data-flow.mdx (via @objectstack/cli)
  • content/docs/api/environment-routing.mdx (via @objectstack/cli)
  • content/docs/api/error-catalog.mdx (via @objectstack/cli)
  • content/docs/automation/hook-bodies.mdx (via packages/cli)
  • content/docs/concepts/metadata-lifecycle.mdx (via @objectstack/objectql)
  • content/docs/data-modeling/formulas.mdx (via packages/objectql)
  • content/docs/deployment/backup-restore.mdx (via @objectstack/cli)
  • content/docs/deployment/cli.mdx (via @objectstack/cli)
  • content/docs/deployment/migration-from-objectql.mdx (via @objectstack/objectql)
  • content/docs/deployment/self-hosting.mdx (via @objectstack/cli)
  • content/docs/deployment/validating-metadata.mdx (via packages/cli)
  • content/docs/deployment/vercel.mdx (via @objectstack/objectql)
  • content/docs/getting-started/your-first-project.mdx (via @objectstack/cli)
  • content/docs/kernel/runtime-services/data-service.mdx (via @objectstack/cli)
  • content/docs/kernel/runtime-services/examples.mdx (via packages/objectql)
  • content/docs/kernel/runtime-services/index.mdx (via packages/cli)
  • content/docs/kernel/services-checklist.mdx (via @objectstack/objectql)
  • content/docs/kernel/services.mdx (via @objectstack/objectql)
  • content/docs/permissions/authentication.mdx (via @objectstack/cli, @objectstack/objectql)
  • content/docs/plugins/index.mdx (via @objectstack/cli, @objectstack/objectql)
  • content/docs/plugins/packages.mdx (via @objectstack/cli, @objectstack/objectql)
  • content/docs/protocol/kernel/index.mdx (via @objectstack/objectql)
  • content/docs/protocol/kernel/plugin-spec.mdx (via @objectstack/cli)
  • content/docs/protocol/kernel/realtime-protocol.mdx (via @objectstack/cli)
  • content/docs/protocol/objectql/query-syntax.mdx (via packages/objectql)
  • content/docs/protocol/objectql/state-machine.mdx (via @objectstack/objectql)
  • content/docs/releases/implementation-status.mdx (via @objectstack/cli, @objectstack/objectql)
  • content/docs/releases/v16.mdx (via @objectstack/cli)
  • content/docs/releases/v17.mdx (via @objectstack/cli)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@github-actions github-actions Bot added size/xl documentation Improvements or additions to documentation tests tooling labels Aug 7, 2026
…g it to any (#6063)

The slot-lookup rule (#4168/#4251) rejects `const engine: any =
getService('objectql')` — the sibling migrate commands are silent only because
they are grandfathered by file, and that baseline only shrinks. The command's
real requirement is `SummaryBackfillEngine` (the slot contract plus the one
member the backfill reads), so name that type: the call site keeps its
checking and the lookup is not erased.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We

Copy link
Copy Markdown
Contributor Author

补记:第二道本地闸门也红过一次并已修(正文「命令与真实输出」只记了第一道)

npx eslint 对新命令壳报 slot-lookup 规则(#4168/#4251):

packages/cli/src/commands/migrate/summary-nulls.ts
  175:13  error  Do not erase a service-lookup result to `any` ...  no-restricted-syntax

来源是照抄兄弟命令的 const engine: any = stack.kernel.getService('objectql') —— 那两个文件之所以安静,是因为它们在 scripts/slot-lookup-baseline.json按文件豁免,而该 baseline 只缩不增,新文件进不去。

修法不是加白名单,而是把这个调用点真正需要的类型写出来:命令需要的是「slot 契约 + 迁移读的那一个成员」,即 SummaryBackfillEngine(objectql 已导出),于是

import type { SummaryBackfillEngine } from '@objectstack/objectql';
...
const engine: SummaryBackfillEngine = stack.kernel.getService('objectql');

import type 在编译期擦除,重的引擎包仍然是下面那句 await import('@objectstack/objectql') 惰性加载的。

复跑(真实输出):

$ npx eslint packages/objectql/src/summary-{aggregate,backfill}.ts \
            packages/objectql/src/summary-backfill.test.ts \
            packages/cli/src/commands/migrate/summary-nulls.ts
ESLINT_EXIT=0

$ pnpm check:slot-lookup
✓ slot-lookup ratchet holds: 143 unswept site(s) in 32 file(s), none new.
  baseline key set verified against 811c30c: no files added.

$ pnpm --filter @objectstack/cli typecheck && pnpm --filter @objectstack/cli build
> tsc --noEmit          # 均无输出即通过

$ pnpm --filter @objectstack/cli test -- --maxWorkers=2
 Test Files  88 passed (88)
      Tests  880 passed (880)

肢 B(注册接线)在这次重建后复验仍成立:node bin/run.js migrate --help 列表含 migrate summary-nulls

两道本地闸门红→绿的记录都留在这里,是因为它们各自指向同一条纪律:grandfather 名单不是静音键,新文件该做的是把类型写对,不是把自己加进豁免。


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

ACCEPT(执行席 PM 验收,冻结期收尾单)

核过:① 三取舍裁定逐条落地 —— 显式 os migrate summary-nulls(dry-run 默认 + --apply,occupancy gate 与证据报告照 files-to-references 先例)、逐行重算(便宜路线被用例 1 钉死,肢 A 还多红出一条同方向更重的证据:写 0 后下次子写入重算把列改回真值,「用户没动、列自己变了」)、引擎层驱动无关(JS 侧判 null 避开 IS NULL 下推的驱动分歧 —— 这个取舍点找得准);② 偏离 1 裁定:采纳 —— 「不触 engine.ts」与「同源同算给代码证据」在本单里真实冲突,dev 选了正确性:纯搬运抽出 summary-aggregate.ts(头注释把三个调用点与漂移后果写成了下一个读者可查证的推导),4 hunk 全在 roll-up 机制内、与在飞 #5351 落点不相邻、合并零冲突、语义零变化由 summary-rollup 全绿 + CI 双向兜底;③ 两道本地闸门中途红的修法都选了「修根因而非加白名单」(去 as any / 写出真类型),留痕完整;④ 幂等、min/max/avg 原样并报告 out of scope、租户形态照家族先例、未记 migration flag 的取舍已文档化;⑤ 必答项四条全有代码级证据。

⚠️ 与在飞 #5351 批的 import 段可能有 hunk 邻接 —— 后落地方按纪律 merge origin/main + merge-tree 预检(其派发令已含)。

翻 ready + auto-merge,进队列(rc.5 口径)。known-issue 文案的「待 #6063 迁移」从此有了具体指令名:os migrate summary-nulls --apply


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/xl tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

summary count/sum 存量 NULL 行的一次性回填 —— #5749 方案 1 落地后的遗留半边(原地升级的库仍漏行)

2 participants