Skip to content

docs(research): add Jev external evidence supplement - #4873

Merged
huangruiteng merged 1 commit into
loopx-project:mainfrom
Duang777:codex/jev-external-evidence-supplement
Sep 22, 2026
Merged

huangruiteng merged 1 commit into
loopx-project:mainfrom
Duang777:codex/jev-external-evidence-supplement

Conversation

@Duang777

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

  • Goal/source and gap: Supplement the optional semantic assistance RFC with external Jev evidence, including dataset scope, metric definitions, selection bias, and implementation boundaries.
  • Observable before → after: The RFC now links to a compact evidence register that distinguishes local classification results from unproven D7/D8 value. D4-shaped screening is optional, and no research or adoption gate changes.
  • Issue/task and intended base: User-requested research supplement; base main. No issue closure.

Scope And Continuation

  • Completed scope and remaining work: Complete within this documentation scope. Adds one Chinese research note, a research index entry, and matching references in both RFC languages.
  • Slice boundary / successor: Model experiments and adoption remain outside this change and subject to the RFC owner decisions. No runtime implementation or new framework is proposed.

Validation

  • Tested revision: 6ce0d02375b00e54bc462b51bcf24cbe4ef44333
  • Run state: finished
  • Input classes: public_fixture
Check kind Result Public-safe evidence / limitation
static passed uv run --no-sync --extra test python examples/docs-governance-smoke.py
static passed uv run --no-sync --extra test python examples/docs-asset-integrity-smoke.py
static passed git diff origin/main...HEAD --check
manual passed Source comparisons address denominators, imported baselines, development-set tuning, and caller-specific inference limits.
real_backend not_run No Jev API calls or independent benchmark reproduction. Published results remain source-reported evidence.
  • Coverage and gaps: Documentation checks cover repository structure and assets. Source citations preserve evidence limitations; this PR makes no claim of LoopX model qualification. Runtime tests are not applicable.

Frontend / Visual Evidence

  • UI impact: none
  • Before / after / states / viewports: Not applicable; Markdown content and references only.
  • Source data: public_fixture
  • Attention review: No documentation chrome or product viewport changes.

Type of Change

  • Documentation update

LoopX Area

  • Public docs or presentation surface (README, protocols, dashboard)

Technical Direction

  • Architecture and research incubator; docs/architecture/rfcs/optional-semantic-assistance-jev-v0.md. RFC decision status and Q1-Q7 remain unchanged.

Shared-authority RFC fixture impact

Not applicable; no shared-authority or TypeScript control-plane changes.

Boundary Checklist

  • Neither the diff nor this PR body discloses private state, credentials, raw traces, internal links, or local machine paths.
  • No maintainer-owned benchmark execution is duplicated.
  • The change stays scoped to the research supplement.
  • UI impact is none.
  • Every commit includes a DCO sign-off.

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact head reviewed: 6ce0d02375b00e54bc462b51bcf24cbe4ef44333

动机

没有发现阻塞问题。这个 PR 补的不是 Jev 的产品结论,而是现有 accepted-for-discussion RFC 缺失的一层可核对外部证据:哪些公开结果只支持“小型固定集合上的局部判断值得研究”,哪些结果受开发集调阈值、外部基线、候选集固定、样本重复或分布变化限制,因此不能被提升为 LoopX 的 D7/D8 资格、产品优先级或运行时 authority。

这个交付边界是成立的。研究证据可以独立审阅、纠错和回滚,而 Q1–Q7 仍由 RFC owner 决定;它没有用“文档已写完”替代研究准入、数据出站、预算、采用或实现决定。

改动思路

改动把快变化的外部材料放在单一、注明日期的非规范 research note 中,只在研究索引和中英文 RFC 的 Appendix C 增加 E6 引用。这样 RFC 继续拥有规范性状态和决策边界,证据快照只负责提供来源、分母、方法限制和可否推出的结论,避免把同一批易过期数据复制进两份规范正文。

我按最强反例审阅了这条结构:如果读者只看到 100% 检出、96.5% 路由或低调用成本,会不会误判为已完成 LoopX-local qualification。当前文本通过逐行限制、jev-harness-lab 汇总/方法章节冲突总量说明、固定 revision 链接,以及 Q1–Q7 的“仍未成立”列,明确阻止了这种推断。

具体改动

关键内容讲解

  1. jev-external-evidence-supplement-v0.zh-CN.md 用一张表登记 TypeSafe、LangChain、SciFact、MetaTool、SkillRetBench、InjecAgent、RouterBench、Who&When、judge-audit 和 Backnotprop 的报告结果,同时把样本规模、调参/选择偏差和不可外推边界放在同一行,避免只保留 headline metric。
  2. 三个实现案例分别证明不同边界:browser-use 只在当前 DOM 候选内选择且执行前重验;compaction 会实际删减结果/调用但保留 pinned/recent 并在失败或压缩不足时回退;auto-mode 先执行确定性 hard-deny,Jev 不可用、取消或返回非法结构时 fail closed。文档没有把三者误抽象成一个已经验证的通用 provider。
  3. Q1–Q7 表把外部证据能带来的研究建议与仍未成立的结论分开;RFC 两种语言只增加 E6,研究索引只增加一个入口,没有改变 M0 状态、里程碑或 authority。

我对材料数字和实现边界做了独立抽查:官方页面确认 jev-1.13.0、每百万 input tokens 0.042 美元以及 64k/32k 限制;LangChain 报告确实只有 5 条固定轨迹、每条 100 次且未记录服务模型版本;固定仓库材料确认 60×15 SciFact、MetaTool 200+200 与 96.5%、SkillRetBench 指标/外部 baseline 读取、1,105/130/825/1,403 分母、120 行但 61 种文本的 ablation、1,688/6,115 两组 guardrail 数据,以及三个实现的失败边界。

对主干的风险

主干风险低且局限于研究文档可能随外部页面漂移。这个风险已通过访问日期、固定 commit 链接、source-reported/未复跑声明和非规范定位约束;任何未来采用仍必须回到目标 caller 的独立留出集、授权数据、冻结对照与 owner 决定。

本地验证:

  • uv run --extra test python examples/docs-governance-smoke.py:通过;
  • uv run --extra test python examples/docs-asset-integrity-smoke.py:通过(6 assets);
  • git diff --check origin/main...HEAD:通过;
  • 公开/私有边界扫描:没有新增本机路径、凭证、内部链接或私人运行材料;
  • 相关 PR/历史扫描:只找到本 Jev RFC 主线及其已合并演进,没有重复的并行 evidence supplement。

未运行 Jev API,也没有独立复现实验;这不是遗漏,因为本 PR 明确交付的是带限制的外部证据登记,而不是模型资格证明。

我的整体评价

APPROVE。这个 exact head 提供了一个完整、可追踪、可回滚的研究增量:材料有来源,headline 有分母,局限与结论并列,且没有把 evidence intake 偷换成 study/adoption approval。代码量为零,106 行全部属于单一研究主题;把内容再塞回双语 RFC 会增加重复和漂移,当前单一 supplement 的放置更合理。

向前看的边界也合适:以后若外部事实变化,只更新有日期的 evidence note;只有 owner 真正接受 Q1–Q4 后,才另开 M1 study。当前无需新增 capability、provider、schema 或通用框架。

English verdict: APPROVE — exact head 6ce0d02375b00e54bc462b51bcf24cbe4ef44333 adds a traceable, caveated external-evidence snapshot without changing the Jev RFC's accepted-for-discussion status, study/adoption gates, data authority, or runtime behavior; the documentation checks and material source spot-checks passed with no blocking finding.

@huangruiteng
huangruiteng merged commit 1eee87f into loopx-project:main Sep 22, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants