Skip to content

feat(web): add bounded transcript content search - #387

Open
testikun wants to merge 3 commits into
openpi-dev:mainfrom
testikun:codex/issue-349-transcript-search
Open

feat(web): add bounded transcript content search#387
testikun wants to merge 3 commits into
openpi-dev:mainfrom
testikun:codex/issue-349-transcript-search

Conversation

@testikun

@testikun testikun commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Problem

Issue #349 requires transcript-aware server search, but the current Web snapshot exposes only a bounded title/cwd/first-message projection. The open metadata-search PRs do not provide a safe mechanism for scanning Session JSONL content or tool evidence.

Relates to #349. This PR implements the transcript-scanner slice only; it does not duplicate PR #362's endpoint or PR #376's pagination work.

Value

  • Gives the Web host a reusable server-side primitive for finding prior user/assistant text and bounded tool evidence.
  • Keeps Pi Session files authoritative while making partial, malformed, changed, unavailable, and budget-limited outcomes observable.
  • Preserves exact Session/message/turn/workspace/source/timestamp provenance without retaining an unbounded index.

Approach

  • Add searchWebTranscripts() over a caller-authorized Session catalog; the scanner never discovers or widens filesystem authority.
  • Enforce hard caps for catalog files, total bytes, per-file bytes, line bytes, retained results, snippet bytes, and wall time; support AbortSignal cancellation.
  • Require regular non-symlink files within canonical allowed roots and exact workspace membership, then validate the Session header identity before publishing matches.
  • Scan JSONL incrementally in fixed-size chunks, retain only a bounded newest-result set, propagate turn identity across intervening Pi entries, and re-check file identity after scanning.
  • Search user/assistant text and tool names only; exclude tool-call arguments/input and tool-result bodies/details. Return control-safe, bidi-safe, UTF-8-bounded plain-text snippets with explicit truncation evidence.
  • Fail soft per file for malformed, changing, deleted, oversized, and unauthorized inputs, with exact partialReasons and accounting.

The existing lightweight metadata filtering remains separate. Endpoint composition, pagination/cursors, and derived index invalidation remain follow-up slices of #349.

Validation

Integrated with main c1c60cd9e357044b53042ab23963d053b46dae9f, preserving the author's commits. The scanner and its test are byte-identical to reviewed head c2f395b; no tool registration, prompt context, or host route is added.

  • bun run check: passed with the frozen lockfile and Pi 0.85.1.
  • bun run test on integrated tree e5af1744e3525d5859929eff2c44c076d5a3f357: Node 1436 passed, 1 skipped, 0 failed; Vitest 30 passed.
  • Transcript-search tests: 9/9 passed, included in the full suite.
  • Independent probes confirmed tool arguments/input/results/details do not enter matches or snippets; tool-name searches return only the name.
  • Earlier member-CLI runs failed on two plan-mode rendering assertions and a setup child timeout. The same failures reproduced on unmodified main; an unchanged-tree serial run from the main review environment passed. The exact environmental trigger was not established; no production or environment workaround was applied.
  • No manual UI or installed-Pi acceptance is claimed. The scanner is not yet connected to a host endpoint or UI.

Impact

  • User-visible behavior: None until a Web endpoint/UI composes the scanner.
  • Model-visible context/tools: None.
  • Runtime/lifecycle: Adds a read-only, cancellable, bounded Session-file scanner; no second Session store or index.
  • Persisted config/data: None.
  • Compatibility/risk: Additive module only. The caller must supply the authorized Session catalog and roots; the scanner fails closed on provenance or path mismatches.

@testikun

testikun commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

CI status note: Node 22/24 completed this PR's repository checks and failed only at the final fresh-install Smoke-test packed standalone Web CLI step, where the child exits before the test can read its URL. The same failure now occurs on unrelated PRs #385, #386, and #387.

I reproduced the packed install independently: npm now admits Pi 0.85.0 through OpenPI's published peerDependencies: "*"; the CLI then exits with Cannot find module '@earendil-works/pi-server'. The checkout/lock-based bun run check and full test suite remain green. I recorded the shared packaging regression and suggested peer-range follow-up in #328: #328 (comment)

No dependency workaround is mixed into this feature PR.

@tt-a1i tt-a1i left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要改:tool-call 参数会进搜索 snippet

web/search/transcript-search.tsmessageEvidencetoolCall 做了:

const payload = tool.arguments === undefined ? tool.input : tool.arguments;
text: `${tool.name} ${JSON.stringify(payload)}`

PR 正文写排除 private result details,但 arguments / input 整段可搜、可进 snippet。工具参数里的 token、路径、密钥会漏到 Web 搜索面。

toolResult 正文也会整段进 index。请把 arguments 从 searchable text 拿掉,或只留 allowlisted 的非敏感字段,并补一条「密钥不得出现在 snippet」的测试。

原语本身(文件/大小/命中上界、symlink 检查、AbortSignal)是好的,泄漏修完再合。现在还没有 HTTP endpoint,但这个函数一旦接上就会爆。

@testikun

testikun commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

已按 review 修复并推送到 c2f395b:toolCall 的 arguments/input 和 toolResult 的正文不再进入搜索 evidence;工具名仍保留为非敏感 provenance。新增回归测试验证参数中的 Bearer token 和结果中的 apiKey 不会出现在搜索命中或 snippet 中。transcript-search 测试 9/9 通过,Biome 和 diff 检查通过。

Keep author history. No feature changes beyond clean automatic integration with c1c60cd.

@tt-a1i tt-a1i left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review of integrated 3e04d85 (tree e5af1744e3525d5859929eff2c44c076d5a3f357). The previous tool-payload indexing blocker is fixed: tool calls/results contribute names only; arguments, input, result bodies and details are excluded. Focused tests and independent probes exercised those exclusions along with path, provenance and resource bounds. No blocking finding was confirmed.

The main integration is clean and preserves the reviewed scanner/test bytes; the delta is only those two new files. No tool registration, model context, persistence writes or host endpoint is changed.

Validation on the integrated tree: bun run check passed; serial bun run test passed (Node 1436 passed, 1 skipped, 0 failed; Vitest 30 passed), including 9 transcript-search tests. Earlier member-CLI full runs had two rendering failures and a setup timeout that also reproduced on pure main; those logs are retained, the exact environment trigger remains unknown, and no workaround was applied to obtain this pass.

This is approval of the scanner primitive, not end-to-end search UI or installed runtime acceptance. Required remote CI and repository approval rules still apply; no merge was performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants