Conversation
Add an in-process memory consolidation mode to `ov compile`. When `--skill` is the sentinel value `memory`, CompileService runs the existing memory framework (ConsolidationExtractContextProvider -> ExtractLoop -> MemoryUpdater) directly inside OpenViking core to dedup/merge/split/compact an existing memory-type directory in place, conforming to that type's schema. The default skill path (VikingBot Wiki compile) is unchanged. Highlights: - New ConsolidationExtractContextProvider: agentic exploration with ls/search/read tools seeded by a recursive listing; single schema inferred from --to; space (self/peer) taken from the canonical --to URI so listing never depends on an empty ctx.user_id. - MemoryCompileRunner: session.commit-lite task shape (task_tracker + one in-process asyncio.Task, no QueueFS re-delivery), bound to a root span so a trace_id is recorded; result reports adds/updates/deletes (file URIs only, memory_diff.json semantics) classified via read_file_contents. - MemoryLsTool: add recursive listing (relative paths, 500-node cap with truncation) and stop hiding subdirectories so subfoldered dirs are not misreported as empty. Only the compile provider exposes ls, so session.commit is unaffected. - CLI: --from optional (required for normal mode, rejected for memory mode); help gains a memory example. - Fix a syntax regression in crates/ragfs/src/lock/provider.rs test module that blocked `make build` (unrelated to compile; introduced by #4908). - Docs: document the memory mode in ov-compile-design.md. - Tests: unit tests for provider/runner/request validation; integration script test_compile_memory_xiaomei.py with merge/split/dedup/preferences cases. Co-authored-by: TRAE CLI <traecli@bytedance.com>
Add a dedicated user-facing page (zh + en) for `ov compile --skill memory` covering when to use it, usage, parameters, behavior, and the adds/updates/ deletes result. Link it from the context-compilation overview. The VitePress sidebar picks the new page up automatically from the directory listing. Co-authored-by: TRAE CLI <traecli@bytedance.com>
e42cde8 to
0589f17
Compare
|
The cuVS failure on head #5177 addresses this existing config-test mismatch: the removed |
chenjw
left a comment
There was a problem hiding this comment.
本次 review 有 2 个需要修复的问题和 4 个建议,详见 inline comments。重点是整理模式的语言解析未接入,以及取消任务后后台仍可能继续执行记忆写入。
| instruction=instruction, | ||
| memory_registry=registry, | ||
| ) | ||
| provider._ctx = ctx |
There was a problem hiding this comment.
[Suggestion] 这里通过 provider._ctx、provider._viking_fs,以及后面的 provider._isolation_handler 从外部补齐 provider 状态。基类构造函数已支持这些依赖,建议让 consolidation provider 显式接收并传递 ctx/viking_fs;需要等待 extract_context 才能创建的 isolation handler 可通过明确的绑定方法设置。这样依赖和初始化顺序更容易检查,也能减少 runner 对基类私有字段的耦合。
| instruction=instruction, | ||
| ctx=ctx, | ||
| ) | ||
| await tracker.complete( |
There was a problem hiding this comment.
[Suggestion] _consolidate() 会把 apply_result.errors 放进结果,但这里始终调用 tracker.complete()。如果所有操作都失败,任务查询仍会得到 completed,仅检查任务状态的调用方会误判成功。建议明确部分成功的语义,至少在零成功且有 errors 时使用 tracker.fail(..., result=result) 保留错误和变更明细,并补一个全部写入失败的测试。
Co-authored-by: TRAE CLI <traecli@bytedance.com>
|
已按 review 处理本轮问题:
本轮暂不处理两个 suggestion:provider 私有字段注入重构、全部 apply error 时将任务改为 failed;前者属于结构清理,后者需要单独明确部分成功语义。 验证:compile/语言/取消测试 15 passed,TaskTracker 并发测试 37 passed;合计 52 passed。Ruff、format、diff check 均通过。 |
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
背景
ov compile原本只有一种模式:用指定的 VikingBot Skill 把来源材料整理成 Wiki 页面。本 PR 新增一种记忆整理模式——把--skill设为哨兵值memory,即可对某个记忆类型目录做就地整理(去重 / 合并 / 拆分 / 改名 / 原地精简),且严格遵守该记忆类型原有的 schema。这条通路完全在 OpenViking 进程内通过现有 memory 框架执行,不经过 VikingBot。默认的 Skill Wiki compile 行为保持不变。用法
--skill memory:哨兵值,触发记忆整理模式(不解析为真实 Skill)。--to:必须是某个记忆类型目录(如.../memories/entities),不能只到.../memories根。--from:记忆模式下不接受;整理就地发生在--to空间内。--instruction:可选,作为整理指令喂给模型;用于点破模型无法自行判断的合并(如“阿珍就是陈静娴”)。返回一个
cmp_前缀的task_id,通过ov task status <task_id>查询。结果含变化文件清单(沿用memory_diff.json语义的adds/updates/deletes+total_*,仅文件 URI 不含内容)与本次整理的trace_id。实现要点
ConsolidationExtractContextProvider:提供ls/search/read,prefetch 用递归 ls 做种子;只加载--to推断出的单一 schema;整理空间由 canonical--toURI 决定。模型只生成该类型的内容操作,但系统可更新其它类型的相邻文件以迁移 links/backlinks。MemoryCompileRunner:使用本地TaskTracker+ 进程内asyncio.Task;任务注册到TaskWorkIndex,ov task cancel会中断实际后台工作。immutable解耦。entities 的category/name使用replace;字段变化时公共MemoryUpdater执行“写新 URI → 迁移 links/backlinks → 删除旧 URI”,结果表现为add(new) + delete(old)。compile 与 session.commit 复用同一逻辑。系统会精确剥离旧 metadata 管理的内部 Markdown href 后按新 URI 重渲染;外部链接和无关手写链接原样保留。跨目录迁移使用 mixed tree lock,最后一个业务文件迁出后删除旧 overview 和空目录。source.delete(replacement=target)合并;源文件关系会继承到目标。MemoryLsTool:支持递归列表(相对路径、500 节点上限 + 截断提示);只在 compile provider 暴露。--from;memory 模式拒绝--from。中英文用户文档说明改名和冲突语义。测试
tests/integration/test_compile_memory_xiaomei.py使用 session.commit 真实抽取数据:person/阿珍.md+person/陈静娴.mdperson/陈静娴.md,删除person/阿珍.md;同步更新关联 eventperson/小林_同事.md+person/小林_教练.mdperson/大壮.md(446 B)person/郑诗晴远.mdperson,文件名用photo_teacher_8616932d.md;保留原事实,不创建重复实体。” ②“把英文人物改回中文名郑诗晴远,同时把目录改成‘人物’;保留原事实。”person/郑诗晴远.md→person/photo_teacher_8616932d.md→人物/郑诗晴远.md→person/photo_teacher_8616932d.md;每步同步更新 3 个关联 eventadd(new) + delete(old),事实保留、旧 raw URI/href 清零,最终空人物/目录删除小美/communication_style.md、food_preference.md、hobbies.md门禁:相关 memory 回归测试 171 passed,mixed-lock 定向测试 3 passed,Rust CLI 3 passed;
ruff check、ruff format --check、git diff --check通过。rename case 已在本地服务完成三步中英文往返:每步均为add(new) + delete(old),关联 event 同步更新,最终空人物/目录返回NOT_FOUND。