Skip to content

feat(compile): 通过 --skill memory 支持记忆整理模式 - #5178

Open
chenjw wants to merge 21 commits into
mainfrom
feat/compile-memory-mode
Open

chenjw wants to merge 21 commits into
mainfrom
feat/compile-memory-mode

Conversation

@chenjw

@chenjw chenjw commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

背景

ov compile 原本只有一种模式:用指定的 VikingBot Skill 把来源材料整理成 Wiki 页面。本 PR 新增一种记忆整理模式——把 --skill 设为哨兵值 memory,即可对某个记忆类型目录做就地整理(去重 / 合并 / 拆分 / 改名 / 原地精简),且严格遵守该记忆类型原有的 schema。这条通路完全在 OpenViking 进程内通过现有 memory 框架执行,不经过 VikingBot。默认的 Skill Wiki compile 行为保持不变。

用法

# 整理 entities(就地去重/合并/规范化)
ov compile \
  --to viking://user/<user_id>/memories/entities \
  --skill memory \
  --instruction "合并明显重复的实体,但不同实体不要合并;保留每个实体的独立事实"

# 也可整理其它记忆类型,例如 preferences
ov compile --to viking://user/<user_id>/memories/preferences --skill memory
  • --skill memory:哨兵值,触发记忆整理模式(不解析为真实 Skill)。
  • --to:必须是某个记忆类型目录(如 .../memories/entities),不能只到 .../memories 根。
  • --from:记忆模式下不接受;整理就地发生在 --to 空间内。
  • --instruction:可选,作为整理指令喂给模型;用于点破模型无法自行判断的合并(如“阿珍就是陈静娴”)。

返回一个 cmp_ 前缀的 task_id,通过 ov task status <task_id> 查询。结果含变化文件清单(沿用 memory_diff.json 语义的 adds/updates/deletes + total_*,仅文件 URI 不含内容)与本次整理的 trace_id

实现要点

  • ConsolidationExtractContextProvider:提供 ls / search / read,prefetch 用递归 ls 做种子;只加载 --to 推断出的单一 schema;整理空间由 canonical --to URI 决定。模型只生成该类型的内容操作,但系统可更新其它类型的相邻文件以迁移 links/backlinks。
  • MemoryCompileRunner:使用本地 TaskTracker + 进程内 asyncio.Task;任务注册到 TaskWorkIndexov task cancel 会中断实际后台工作。
  • 输出语言:在首轮 prompt/schema 构建前解析;显式 override 优先,否则从已有记忆采样检测。
  • URI 迁移:schema 中参与 URI 的字段与 immutable 解耦。entities 的 category / name 使用 replace;字段变化时公共 MemoryUpdater 执行“写新 URI → 迁移 links/backlinks → 删除旧 URI”,结果表现为 add(new) + delete(old)。compile 与 session.commit 复用同一逻辑。系统会精确剥离旧 metadata 管理的内部 Markdown href 后按新 URI 重渲染;外部链接和无关手写链接原样保留。跨目录迁移使用 mixed tree lock,最后一个业务文件迁出后删除旧 overview 和空目录。
  • 冲突策略:新 URI 已存在时明确报冲突、不覆盖。ExtractLoop 自动读取目标后,模型必须显式更新 canonical target,并用 source.delete(replacement=target) 合并;源文件关系会继承到目标。
  • 失败保护:目标写入或关系迁移失败时不删除源文件;compile 和 streaming/session.commit 写入均覆盖源、目标及关系邻居的路径锁。
  • MemoryLsTool:支持递归列表(相对路径、500 节点上限 + 截断提示);只在 compile provider 暴露。
  • CLI 与文档:普通 compile 仍要求 --from;memory 模式拒绝 --from。中英文用户文档说明改名和冲突语义。

测试

tests/integration/test_compile_memory_xiaomei.py 使用 session.commit 真实抽取数据:

场景 输入文件 compile instruction 输出文件 / 变化 结果
merge person/阿珍.md + person/陈静娴.md “阿珍”和“陈静娴”是同一人;规范名用“陈静娴”,保留大学室友、借钱、UI 设计师、下月来访等事实;其它实体不要合并。 保留并更新 person/陈静娴.md,删除 person/阿珍.md;同步更新关联 event 合并成功,独立事实均保留
split person/小林_同事.md + person/小林_教练.md “小林”是两个不同的人:市场部同事和健身教练;拆成可区分的两个实体,不得合并。 文件不变 抽取阶段已正确拆分,compile 合理 no-op
dedup person/大壮.md(446 B) 在不丢失独立事实的前提下,原地精简“大壮”的重复、啰嗦和同义表述;不要修改其它实体。 原 URI 更新为 408 B 去除重复表述,独立事实保留
rename person/郑诗晴远.md ①“把人物郑诗晴远的目录和文件名改成英文:目录用 person,文件名用 photo_teacher_8616932d.md;保留原事实,不创建重复实体。” ②“把英文人物改回中文名郑诗晴远,同时把目录改成‘人物’;保留原事实。” person/郑诗晴远.mdperson/photo_teacher_8616932d.md人物/郑诗晴远.mdperson/photo_teacher_8616932d.md;每步同步更新 3 个关联 event 三步均为 add(new) + delete(old),事实保留、旧 raw URI/href 清零,最终空 人物/ 目录删除
preferences 小美/communication_style.mdfood_preference.mdhobbies.md 合并明显重复或矛盾的偏好,保留每条独立事实;不同维度不要强行合并。 文件不变 内容已无明显重复,compile 合理 no-op

门禁:相关 memory 回归测试 171 passed,mixed-lock 定向测试 3 passed,Rust CLI 3 passed;ruff checkruff format --checkgit diff --check 通过。rename case 已在本地服务完成三步中英文往返:每步均为 add(new) + delete(old),关联 event 同步更新,最终空 人物/ 目录返回 NOT_FOUND

Add an in-process memory consolidation mode to `ov compile`. When `--skill`
is the sentinel value `memory`, CompileService runs the existing memory
framework (ConsolidationExtractContextProvider -> ExtractLoop -> MemoryUpdater)
directly inside OpenViking core to dedup/merge/split/compact an existing
memory-type directory in place, conforming to that type's schema. The default
skill path (VikingBot Wiki compile) is unchanged.

Highlights:
- New ConsolidationExtractContextProvider: agentic exploration with ls/search/read
  tools seeded by a recursive listing; single schema inferred from --to; space
  (self/peer) taken from the canonical --to URI so listing never depends on an
  empty ctx.user_id.
- MemoryCompileRunner: session.commit-lite task shape (task_tracker + one
  in-process asyncio.Task, no QueueFS re-delivery), bound to a root span so a
  trace_id is recorded; result reports adds/updates/deletes (file URIs only,
  memory_diff.json semantics) classified via read_file_contents.
- MemoryLsTool: add recursive listing (relative paths, 500-node cap with
  truncation) and stop hiding subdirectories so subfoldered dirs are not
  misreported as empty. Only the compile provider exposes ls, so session.commit
  is unaffected.
- CLI: --from optional (required for normal mode, rejected for memory mode);
  help gains a memory example.
- Fix a syntax regression in crates/ragfs/src/lock/provider.rs test module that
  blocked `make build` (unrelated to compile; introduced by #4908).
- Docs: document the memory mode in ov-compile-design.md.
- Tests: unit tests for provider/runner/request validation; integration script
  test_compile_memory_xiaomei.py with merge/split/dedup/preferences cases.

Co-authored-by: TRAE CLI <traecli@bytedance.com>
Add a dedicated user-facing page (zh + en) for `ov compile --skill memory`
covering when to use it, usage, parameters, behavior, and the adds/updates/
deletes result. Link it from the context-compilation overview. The VitePress
sidebar picks the new page up automatically from the directory listing.

Co-authored-by: TRAE CLI <traecli@bytedance.com>
@chenjw
chenjw force-pushed the feat/compile-memory-mode branch from e42cde8 to 0589f17 Compare September 18, 2026 18:20
@r266-tech

Copy link
Copy Markdown
Contributor

The cuVS failure on head 0589f178 is the retired-option assertion in test_cuvs_filter_cache_rejects_negative_size (106 passed, 1 failed), rather than a memory-compile assertion. The API/CLI integration check passed.

#5177 addresses this existing config-test mismatch: the removed cuvs_filter_cache_size option is ignored under the config's current unknown-field compatibility policy. Its cuVS and API/CLI checks both passed on b500a354. Once that fix lands, updating this branch should remove this particular failure; the current check here is still failing.

@chenjw chenjw left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本次 review 有 2 个需要修复的问题和 4 个建议,详见 inline comments。重点是整理模式的语言解析未接入,以及取消任务后后台仍可能继续执行记忆写入。

Comment thread openviking/session/memory/consolidation_context_provider.py Outdated
Comment thread openviking/service/memory_compile.py
Comment thread openviking/session/memory/consolidation_context_provider.py Outdated
Comment thread docs/design/ov-compile-design.md
instruction=instruction,
memory_registry=registry,
)
provider._ctx = ctx

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] 这里通过 provider._ctxprovider._viking_fs,以及后面的 provider._isolation_handler 从外部补齐 provider 状态。基类构造函数已支持这些依赖,建议让 consolidation provider 显式接收并传递 ctx/viking_fs;需要等待 extract_context 才能创建的 isolation handler 可通过明确的绑定方法设置。这样依赖和初始化顺序更容易检查,也能减少 runner 对基类私有字段的耦合。

instruction=instruction,
ctx=ctx,
)
await tracker.complete(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] _consolidate() 会把 apply_result.errors 放进结果,但这里始终调用 tracker.complete()。如果所有操作都失败,任务查询仍会得到 completed,仅检查任务状态的调用方会误判成功。建议明确部分成功的语义,至少在零成功且有 errors 时使用 tracker.fail(..., result=result) 保留错误和变更明细,并补一个全部写入失败的测试。

Co-authored-by: TRAE CLI <traecli@bytedance.com>
@chenjw

chenjw commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

已按 review 处理本轮问题:

  • 修复输出语言:在 ExtractLoop 构建首轮 prompt/schema 前解析语言;显式 output_language_override 优先,否则从目标目录最多采样 3 个记忆文件(每个 4 KiB)检测语言。采样不进入 read_file_contents,不绕过修改前完整读取保护。补充中文记忆与语言 override 回归测试。
  • 修复取消语义:MemoryCompileRunner 将当前 asyncio task 注册到 TaskTracker,并用 bind_task_context 绑定后续队列工作;取消会中断后台整理,最终进入 cancelled,且不会执行 MemoryUpdater 写入。补充取消期间不写入的回归测试。
  • 删除 consolidation provider 中未使用的 prefetched_uris_raw_ls_list_memory_files,同步修正文档说明。
  • 调整 ov-compile-design.md 章节顺序为 2.1 / 2.2 / 2.3。

本轮暂不处理两个 suggestion:provider 私有字段注入重构、全部 apply error 时将任务改为 failed;前者属于结构清理,后者需要单独明确部分成功语义。

验证:compile/语言/取消测试 15 passed,TaskTracker 并发测试 37 passed;合计 52 passed。Ruff、format、diff check 均通过。

chenjw and others added 18 commits September 22, 2026 02:20
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants