Skip to content

Derive the four telemetry pull requests from the landed recorder - #5347

Merged
lidge-jun merged 11 commits into
devfrom
codex/260920-r5-telemetry-derived
Sep 20, 2026
Merged

lidge-jun merged 11 commits into
devfrom
codex/260920-r5-telemetry-derived

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Summary

The four telemetry pull requests lane C deferred "as implemented", rebuilt as consumers of the
recorder that #5266 and #5300 landed. Reimplements #2366, #3748 and #3983; carries the size-limit
half of #5063. No new store: the durable shapes stay PersistedUsageAttempt and
PersistedUsageEntry, and every projection reads them.

Why a failed request is now attributable (#2366). failureStage and failureCause ride the
attempt that ended a request and the logical row, both closed roster members. The PR's own
FailureSide and seven-member FailureStage are not here — two attribution vocabularies for one
question is the class that blocked 2.60.0 — and its widening of transportPhase and
terminalSource to arbitrary strings is not here either. terminalStatus, which was a plain
string, joins the closed validators beside it, because it is now a grouping-key slot assembled
from an upstream frame. The derivation reads only closed values; errorCode and upstreamError are
excluded because both carry upstream text, so a classification keyed on them is a different answer
per provider and per locale. That exclusion is what lets the pair be a Prometheus label and a
fingerprint component with no masking pass. The resend verdict they imply is never stored
/api/logs computes it at read time, so a row written by an older build cannot assert a permission
the current tables refuse.

Why recurring failures group without a second ledger (#3748). src/telemetry/ and its SQLite
store are not built. Failed rows are folded during the existing cooperative ledger scan into a
versioned fingerprint over a fixed-arity tuple of closed members, inheriting that scanner's 1 MiB
row ceiling, chunk size, checkpoint identity and boundary digest. The free-text signature and its
regex masking are replaced by construction rather than by a better regex: an expression can only
assert it removed what it matched, while a tuple whose every slot comes from a frozen list has
nothing to remove. Read with GET /api/usage?failures=1, opt-in because it costs a scan.

Why stream diagnostics need no second emission path (#3983). emitDebugLine writes the ring
and stderr, and a service manager redirects stderr to a file, so the PR's per-event lines would
give an installed service a durable per-event history beside the ledger; its per-payload HMAC used a
process-global key, making every repeated prompt fragment and tool name correlatable for the process
lifetime. Five bounded counts on the attempt answer the same questions and cannot carry content.
Adapter events are counted at the adapter-parse seam and relayed frames after a successful enqueue —
counting both at the reader would make them equal by construction and erase the loss signal. The
debug ring now formats one line per finalized attempt from those counts.

Why retention can delete rows safely (#5063). usageLedgerMaxBytes is unset by default and
unset means unlimited. The defect this closes: #5063 captured a size, copied a suffix and renamed
over whatever was there, so a row appended in between was silently dropped. Now the compaction runs
inside the synchronous append call stack, a second server on the same home cannot append at all
because the existing ledger-owner lease refuses it, and validateBeforeRename re-opens the target
immediately before the rename and refuses unless identity, size and revision metadata are
byte-for-byte what was copied. Rows are copied and never parsed, which is what keeps a field a newer
build wrote intact through a compaction. A compaction also discards the Logs ring, the retained
aggregates and the request-history index — otherwise /api/logs keeps serving rows the ledger no
longer has.

Before/after, concretely: a turn whose upstream stream died mid-flight was a green row with a bare
upstream_server_error; it is now transport-ambiguous at protocol-prelude, reported identically
by the dashboard, the durable row and opencodex_request_failures_total, with a resend verdict that
names which refusal it is.

Closes #2366
Closes #3983

#3748's grouping is built and readable, but its operator surface is one query parameter rather than
a page; the coordinator decides whether that closes it.

#5063 is deliberately not closed. Its Usage-page control is not here. This lane may not build or
run the GUI, so it cannot produce the screenshot that gate requires, and shipping an unverifiable
control is worse than shipping the policy it would set. Remaining scope: the dashboard control, its
management route, and the ten catalog strings. The limit is settable in config.json today and the
configuration reference documents it.

Verification

Static source review plus exact-head hosted CI, and four high-effort adversarial reviews — one per
carried pull request before implementation, then three over the finished branch covering typecheck
hazards, repository gates, and runtime correctness and privacy. Their findings are in the branch:
transport evidence now outranks a synthetic status, 402 maps to quota-exhausted,
transport-unsent is no longer the fall-through (it permits a resend, so it is reachable only from
a site that can prove a pre-connect failure), the streamed writer fsyncs the parent directory,
buffered responses no longer read as total relay loss, two tests read rosters instead of restating
them, and the invariant is split so each of INV-RESEND-01 and INV-ATTRIBUTION-01 binds exactly one
test.

Checked statically on this branch:

  • no touched file is at or over its file-size ratchet cap; src/server/index.ts sits at 884 against
    893, and the ten locale catalogs are exempt;
  • scripts/test-layout/layout.json and tests/fixtures/test-layout-expected.json agree key for key,
    and each new test's regex seed resolves to the domain it is registered to;
  • src/usage/telemetry-contract.ts still has no imports, src/usage/request-outcome.ts still reaches
    nothing but it, and gui/src/pages/Logs.tsx still never names src/usage/log;
  • all 29 new label keys exist in all ten catalogs and those edits are purely additive;
  • no test or document restates a source constant.

NOT RUN on this branch, by instruction, and not to be recorded as passing: bun run test, any
individual bun test file, bun run typecheck, bun run build:gui, bun run lint:gui,
bun install, bun run structure:check, bun run privacy:scan, and any live ocx execution.

The GUI screenshot gate

missing_ui_screenshot fires, and it is a fair gate here: this adds UI, not only strings. What
follows is the evidence for a judgement, not an argument for a waiver.

What is new on screen. Three rows in the Logs detail dialog's Basic section, and one changed
value in the attempt table:

Where Before After
Detail → Basic Time, Outcome, Upstream sends, Request, … adds Failure cause (with the stage it reached, inline in parentheses) and Resend, both only when the row failed
Detail → Attempts, "Reason" column the raw wire errorCode, e.g. upstream_server_error the localized cause, e.g. "Upstream fault"; errorCode remains the next fallback

Why the layout risk is low, stated precisely. Both new rows are label/value pairs inside the
existing .log-detail-grid, which is a two-column max-content minmax(0, 1fr) grid that already
renders Time, Outcome and Upstream sends the same way. No CSS is added or changed, no new class
appears, no column is added to any table, and nothing becomes conditional that was not already —
the adjacent Upstream sends row uses the identical {cond && (<>…</>)} pattern. The only genuinely
new visual is a longer value string in the second grid column, which that column already wraps via
minmax(0, 1fr).

What a screenshot would still add. Text length. The longest English value is "Not resent: the
same request would fail again", and German and Vietnamese are longer still; at the 7rem label
column the narrow breakpoint uses, that wraps. Wrapping is the expected behaviour of this grid, but
"expected" is a source claim, and only a capture shows whether it reads well next to the row above
it.

What source checking already settles. The ten catalog edits are +29 lines and 0 removed in each
file, every one of the 29 keys exists in all ten catalogs, and three satisfies clauses make a
missing label a typecheck failure rather than the silent fallback that previously rendered four real
recovery kinds as "Unknown recovery reason".

Two limits this branch does not close

Only the attempt that ends a request, plus the one sealed by a key-account rotation, carry
attribution; six intermediate finalizers in policy-fallback.ts and core-combo.ts still reach the
ledger unattributed. Each has different evidence in scope, and a branch verified by static review
alone should not add six classification call sites at once. The logical row is attributed in every
case, which is what the projection and the exporter read.

A ciphertext or reasoning-parameter recovery that succeeded, followed by an unrelated 400 on the same
attempt, still reads as that recovery's cause. The rule is narrowed to the last recorded kind on the
matching status; the proper fix belongs in the recovery path.

Both are recorded in devlog/_plan/260920_round2_followups/050_lane_r5.md, along with a pre-existing
defect found while reviewing the atomic writer: its scrub fallback opens with "wx" and so always
fails on an existing temp. That one predates this branch and sits on a security-adjacent path, so it
is left for its own change rather than fixed inside a telemetry branch.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • New Features

    • Request logs now show failure causes, processing stages, and whether a request can be safely resent.
    • Usage reporting can group failed requests by recurring failure patterns through the optional failures view.
    • Added optional usage-history size limits that retain the newest complete records while preserving data integrity.
  • Localization

    • Added translated failure details and resend-status labels across supported languages.
  • Documentation

    • Documented usage-history size limits, retention behavior, minimum values, and current configuration limitations.

@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner September 20, 2026 13:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-20T16:14:21.808375Z 9ea3974 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: c0b13a6c-bd55-45ed-98e4-be4f3af70b45

📥 Commits

Reviewing files that changed from the base of the PR and between 91380c7 and 714ddbf.

📒 Files selected for processing (51)
  • devlog/_plan/260920_round2_followups/050_lane_r5.md
  • docs-site/src/content/docs/reference/configuration/server.md
  • docs-site/src/content/docs/reference/management-api.md
  • gui/src/i18n/de.ts
  • gui/src/i18n/en.ts
  • gui/src/i18n/fr.ts
  • gui/src/i18n/ja.ts
  • gui/src/i18n/ko.ts
  • gui/src/i18n/ru.ts
  • gui/src/i18n/tr.ts
  • gui/src/i18n/vi.ts
  • gui/src/i18n/zh-TW.ts
  • gui/src/i18n/zh.ts
  • gui/src/pages/Logs.tsx
  • scripts/test-layout/layout.json
  • src/bridge/response-json.ts
  • src/bridge/sse.ts
  • src/config/atomic-write.ts
  • src/config/schema/config-schema.ts
  • src/lib/debug.ts
  • src/lib/request-failure-attribution.ts
  • src/lib/request-failure-model.ts
  • src/server/index.ts
  • src/server/management/logs-usage-routes.ts
  • src/server/management/shared.ts
  • src/server/request-log-failure-attribution.ts
  • src/server/request-log.ts
  • src/server/request-metrics.ts
  • src/server/responses/request-transport.ts
  • src/server/usage-ledger-retention.ts
  • src/types/config.ts
  • src/usage/attempt-delivery.ts
  • src/usage/failure-fingerprint.ts
  • src/usage/failure-projection-cache.ts
  • src/usage/failure-projection.ts
  • src/usage/ledger-retention.ts
  • src/usage/log.ts
  • src/usage/request-outcome.ts
  • src/usage/retention-contract.ts
  • src/usage/telemetry-contract.ts
  • structure/gui-and-management-api.md
  • structure/overview.md
  • tests/fixtures/test-layout-expected.json
  • tests/lib/failure-attribution.test.ts
  • tests/server/management-metrics-export.test.ts
  • tests/usage/request-outcome-agreement.test.ts
  • tests/usage/usage-attempt-delivery.test.ts
  • tests/usage/usage-failure-fingerprint.test.ts
  • tests/usage/usage-failure-projection.test.ts
  • tests/usage/usage-ledger-retention.test.ts
  • tests/windows/windows-secret-acl.test.ts
 _____________________________________________________________
< 🥕 I brought carrots. You brought corner cases. Let's party. >
 -------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Deterministic PR hygiene checks passed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dea80a9cb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread gui/src/pages/Logs.tsx
Comment on lines +31 to +32
import type {
AttemptRecoveryKind,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the import compatible with the source-oracle test

Changing this import to a multiline form breaks the existing assertion at tests/usage/request-outcome-agreement.test.ts:194, which searches for the literal substring import type { AttemptRecoveryKind. Running that focused test at this commit fails on this assertion, so either preserve the single-line import prefix or update the oracle to recognize multiline imports. This source-as-data test also requires explicit coverage because import-graph selection cannot discover it.

AGENTS.md reference: AGENTS.md:L225-L230

Useful? React with 👍 / 👎.

Comment thread gui/src/pages/Logs.tsx
Comment on lines +1093 to +1098
{detailFailure.cause && (
<>
<span className="muted">{t("logs.detail.cause.label")}</span>
<span>
{detailFailure.cause}
{detailFailure.stage && ` (${t("logs.detail.stage.label")}: ${detailFailure.stage})`}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Render a failure stage even when no cause exists

For an incomplete request, deriveRequestFailureAttribution intentionally persists failureStage while deriveRequestFailureCause returns no cause, but this condition hides the entire stage display whenever detailFailure.cause is absent. Consequently incomplete turns carry stage telemetry through the management API yet the dashboard never shows it; render the stage independently or gate the block on either field.

AGENTS.md reference: gui/AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Comment on lines +65 to +66
const index = buffer.subarray(0, read).indexOf(0x0a);
if (index >= 0) return position + index + 1;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the row when the cutoff is already a boundary

When captured.size - target is exactly the start of a row, the byte immediately before from is already an LF, but this search ignores that boundary and returns the LF after the row, unnecessarily deleting the first whole row that fits in the retention target. Check whether from is zero or preceded by LF before scanning forward; otherwise the implementation contradicts the documented newest-whole-rows contract in structure/gui-and-management-api.md:542-545.

AGENTS.md reference: structure/AGENTS.md:L9-L10

Useful? React with 👍 / 👎.

@lidge-jun

Copy link
Copy Markdown
Owner Author

리뷰 · 우선순위 56 / 80

이 PR은 예전에 따로 열려 있던 텔레메트리 네 갈래(#2366·#3748·#3983·#5063의 크기 제한 절반)를, 이미 들어온 사용량 recorder 위에서만 다시 만든 것입니다. 새 SQLite·새 타임라인 저장소는 없습니다. 실패한 요청은 닫힌 목록의 failureStage/failureCause로만 원인을 붙이고, 같은 실패끼리 묶는 지문은 그 닫힌 칸들로만 만듭니다. 다시 보내도 되는지(resendPermission)는 디스크에 안 넣고 /api/logs가 읽을 때 계산합니다. 스트림 진단은 내용이 아닌 시도별 카운트 다섯 개로 답하고, usageLedgerMaxBytes가 켜져 있으면 덧붙이는 그 안에서 바이트 복사·이름 바꾸기 직전 검사로 줄입니다. 베이스는 dev입니다. GUI 스크린샷 게이트와 #5063 Usage 페이지 조작은 이 PR이 일부러 안 닫습니다. types/config의 usageLedgerMaxBytes는 추가만입니다.

라인 - tests/usage/request-outcome-agreement.test.tsimport type { AttemptRecoveryKind 검사: gui/src/pages/Logs.tsx는 이제 여러 줄 import type { / AttemptRecoveryKind, 형태입니다. 한 줄 문자열과 안 맞아서 이 테스트는 지금 헤드에서 깨집니다. 본문도 로컬 테스트를 안 돌렸다고 적혀 있습니다. CI test가 돌면 여기서 막힐 가능성이 큽니다.
라인 - src/server/request-log.ts 최종 귀속: outputObserved는 요청 전체의 logCtx.firstOutputMs를 보는데, sideEffectObserved는 마지막 activeAttempt.deliverySummary만 봅니다. 앞 시도에서 도구 호출을 내보낸 뒤 넘긴 요청이면 부작용이 빠진 채 단계가 낮아지고, 다시 보내기 거절(refused-committed)이 약해질 수 있습니다. 출력 쪽 주석이 말한 “안전한 쪽으로 틀어라”와 같은 기준이 부작용에는 안 적용된 상태입니다.
라인 - src/server/usage-ledger-retention.tsdiscardRetainedUsageAggregate(): 주석은 유지 집계를 비운다고 하지만, 기존 함수는 가장 오래된 항목 하나만 떼는 메모리 압박용 API입니다. discardRetainedFailureProjection()은 통째로 null인데 집계 쪽만 다릅니다. 보통은 다음 읽기에서 다시 만들지만, 주석·동작이 어긋납니다.
라인 - src/usage/failure-projection-cache.ts: 줄이는 동안 돌고 있던 스캔이 끝난 뒤 옛 그룹을 다시 retained에 올릴 수 있습니다. 세대/에포크 가드가 없습니다. 또한 이 캐시는 앱 소유 메모리 명단에 없어, 지문이 많으면 압박 시 안 비워질 수 있습니다.
라인 - 본문이 이미 적은 한계도 그대로입니다. 중간 finalizer 여섯 곳은 시도에 원인이 안 붙고, 암호문 복구 성공 뒤 같은 시도의 다른 400은 복구 원인으로 남을 수 있습니다. 이 브랜치 범위 밖이라고 적어 둔 점은 맞습니다.

메인테이너의 판단이 필요한 지점

이 PR이 머지되면 초안 #2366·#3983은 Closes로 닫히고, #3748은 “조회 파라미터만 있고 전용 페이지는 없다”고 조율자 판단에 맡깁니다. #3748을 이걸로 닫을지, 운영자 UI를 남길지 정해 주세요. #5063은 Usage 페이지·관리 라우트·카탈로그가 없어 일부러 안 닫습니다. 동의합니다.

드래프트 #5305(usage.jsonl 고정 64MiB·단일 아카이브 회전)는 이 PR의 선택적 usageLedgerMaxBytes+개정 안전 축소와 같은 파일을 두고 싸웁니다. #5305는 대체·닫기 후보로 보입니다.

gui/와 로케일 열이 바뀌어 missing_ui_screenshot이 뜹니다. gui-screenshot-waived 라벨이나 메인테이너 면제 댓글이 문서상 해법입니다. 보이는 변화는 실패 상세의 원인·단계·재전송 판정 세 줄입니다.

너의 추천

한 줄 import 문자열 검사를 여러 줄 import에도 맞게 고친 뒤에만 머지하세요. 가능하면 최종 귀속의 sideEffectObserved도 요청 범위로 맞추세요(앞 시도 합산 또는 요청 단위 플래그). 집계 discard·프로젝션 in-flight는 주석과 맞추거나 짧게 고쳐 두면 좋습니다. 방향·프라이버시(닫힌 어휘, 원인 키에 upstream 문구 배제, 바이트 복사 축소)는 좋고, types/config 분할·중복 tip으로 이 PR을 닫을 이유는 없습니다. 머지 후 #5305는 닫거나 대체 이유를 한 줄로 남기세요.

이 댓글은 grok-bot이 작성했습니다

@lidge-jun
lidge-jun force-pushed the codex/260920-r5-telemetry-derived branch from dea80a9 to 45dca12 Compare September 20, 2026 13:51
@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

⏳ DRAFT

  • UI screenshot required.

What to do

  • Add a screenshot of the UI change to the PR description.

Automatic draft conversion failed (token cannot change draft status). Please convert this pull request to a draft manually. The required enforce-target check will keep failing until every issue above is resolved.

@github-actions
github-actions Bot marked this pull request as draft September 20, 2026 14:18
lidge-jun and others added 10 commits September 21, 2026 00:23
#2366 asked for durable failure attribution and shipped its own FailureSide and
seven-member FailureStage to carry it. Lane C2 took the rehydration half and left
that vocabulary behind, because defining a second one beside the stage and cause
model that had just landed is the class of defect that blocked 2.60.0. This is the
same answer expressed in the landed vocabulary.

PersistedUsageAttempt and PersistedUsageEntry now carry failureStage and
failureCause, both closed roster members. The resend verdict they imply is NOT
stored: it is derived at read time, so a row written by an older build can never
carry a verdict the current table would no longer reach.

The derivation reads only closed values -- an HTTP status, a terminal status, a
close reason, a transport phase, a recovery kind. errorCode and upstreamError are
deliberately excluded: both are assembled partly from upstream text, so a
classification keyed on them is a different answer per provider and per locale, and
a grouping key built from them cannot promise it carries no content. That exclusion
is what lets the pair be a Prometheus label and a fingerprint component without a
masking pass.

It runs at addFinalRequestLog, the one seam every request passes exactly once
whatever transport served it, and before the attempt snapshot, so the row that
reaches disk and the live attempt object carry the same pair. addRequestLog rebuilds
the persisted row field by field rather than spreading it, so the pair is written
there explicitly -- a field omitted at that line reaches /api/logs and never reaches
usage.jsonl, which is the surface the derived projection reads.

The stage and cause rosters move to src/usage/telemetry-contract.ts and
src/lib/request-failure-model.ts re-exports them, the same relocation lane C2 made
for the recovery roster and for the same reason: the dashboard renders a label per
member, and a type-only import of the table module would drag its import graph into
the browser project. The decision tables stay where they were.

The test runs over a cross product built from the rosters themselves rather than a
written-out list, so a member added later widens the space instead of leaving a case
nobody wrote.

Co-authored-by: chilung <b0423031@gmail.com>
Completes the agreement condition for the attribution the previous commit
records. The durable row carried a cause and nothing showed it, which is the same
shape as the defect lane C2 fixed: a real cause reaching the operator as an
absence of one.

The exporter gains opencodex_request_failures_total{protocol,cause}. It counts the
value the recorder derived rather than deriving one of its own, because the
recorder is the only place that sees the transport facts a cause needs, and two
derivations of one answer is exactly the disagreement this batch exists to remove.
The label set IS the shared dictionary rather than a copy of it. Cardinality is
fifteen causes across four protocols -- sixty series, fixed for the lifetime of the
roster, every value from a frozen list -- and it labels a counter, never a
histogram; a case asserts both.

/api/logs computes resendPermission at read time for the row and for each attempt.
It is never stored: the tables that decide it live in this build, and a row written
by an older one must not assert a permission the current tables would refuse. A
case asserts the pair is in the ledger module and the verdict is not.

The Logs detail dialog shows the cause, the stage it reached and the resend verdict,
and the attempt table leads its reason column with the cause, keeping the exact wire
errorCode behind it because that is what a bug report needs. Three satisfies clauses
make a missing label a typecheck failure rather than a silent fallback, and the
existing catalog oracle now covers the new key groups.

This trips the missing_ui_screenshot gate. This lane may not build or run the GUI,
so it cannot produce the screenshot; the gate fires on changed paths under gui/,
not on words in the description. The visible change is three rows added to the
detail dialog for a failed request and a named cause where the attempt table
previously showed a bare wire code.

Co-authored-by: chilung <b0423031@gmail.com>
…tore

#3748 proposed a privacy-safe failure ledger and built it as a second SQLite store
beside usage.jsonl, keyed by a free-text signature that regular expressions tried
to mask. Both halves are replaced.

The store becomes a projection rebuilt from the canonical ledger. It holds a count
and two timestamps per group and nothing else, so deleting a row from usage.jsonl
removes it from this grouping on the next rebuild -- which is what it means for
retention to have one owner instead of four. It reads through the existing
scanUsageLedgerCooperatively and therefore inherits every bound that scanner
already enforces: the 1 MiB row ceiling, the 1 MiB chunk, the cooperative yield,
the opened-EOF snapshot boundary, and the path/device/inode/birthtime identity with
its 64 KiB boundary digest. A same-size file whose revision metadata moved forces a
rebuild rather than an append, so a replaced ledger can never extend stale groups.

The masked signature becomes a fixed-arity tuple of closed roster members. A regular
expression can only assert that it removed what it matched; a tuple whose every slot
is a member of a frozen list has nothing to remove. The input type cannot express a
model, an account, an error message, a prompt, a request id or a timestamp, so no
amount of upstream text can reach a fingerprint. Absent facts are explicit nulls in
fixed positions, because omitting them would let [a, null, b] and [a, b] collide.

The configured provider name is the one input that starts as free text -- users name
their own provider entries -- so it is resolved against the provider registry and
becomes null when it is not a registry member. A provider named after its owner
groups under null, which is the honest answer.

This exposed a real hole the fingerprint would otherwise have inherited:
terminalStatus was persisted as a plain string and copied through the normalizer on
truthiness alone, unlike the inbound protocol, transport phase and terminal source
beside it. Harmless while it was only rendered; not harmless as a grouping-key slot,
because the value is assembled from an upstream terminal frame. It is now the closed
type, derived from the outcome roster rather than restated, and validated on read
back.

Two parts of the original are deliberately absent. The occurrence list is a second
copy of history with its own retention policy. The mutable
monitoring/dispatched/fixed/ignored status and its notes are operator state, which
cannot be reconstructed from immutable request rows; presenting them as a derived
ledger would be presenting a claim this projection cannot make. They need their own
owner, keyed by the fingerprint, if they are wanted.

The reader is GET /api/usage?failures=1 rather than a new route: it answers a
different question from the usage summary and costs a scan, so it is opt-in and a
dashboard asking for spend does not pay for it.

Co-authored-by: SB Yoon <44089734+yansigit@users.noreply.github.com>
#3983 wanted the signals a stream diagnostic gives -- a missing terminal,
adapter-to-client loss, empty output, partial output size -- and emitted one debug
line per event to get them. Two things make that the wrong shape.

It is a second durable record. emitDebugLine writes the in-process ring AND stderr,
and stderr is redirected to the service log under both launchd and systemd, so an
installed service accumulates a per-event history beside the ledger with its own
retention, sequencing, request identity and masking. And per-event lines needed a
per-payload fingerprint to correlate; under a process-global random key that makes
every repeated prompt fragment, tool name and error message correlatable for the
lifetime of the process.

Five bounded counts on PersistedUsageAttempt answer the same questions and cannot
carry content at all. They ride the attempt, so they inherit the ledger's
normalization, masking and retention instead of acquiring their own, and the debug
ring now FORMATS one line per finalized attempt from what the recorder already
counted -- appendDebugLogLine directly, never emitDebugLine, so the ring is a live
view of the durable record rather than a parallel source for it.

The counting point matters. Adapter events are counted at the one seam every adapter
parse already passes; relayed frames are counted after a SUCCESSFUL controller
enqueue in the SSE bridge. Counting both at the reader would make the two numbers
equal by construction and erase the one discrepancy they exist to expose. The
recorder is bound to the request's translator budget -- an object every bridge on
the delivery path already receives -- and reaches the current attempt through a
callback rather than holding one, so a mid-request attempt rotation credits the
attempt that is live rather than one already finalized.

sideEffectEvents feeds the failure stage, which makes side-effect reachable for the
first time: a relayed tool call is an externally visible effect, so the resend
verdict refuses. Counting it at the transport rather than the adapter is what makes
that correct -- an emitted tool call the client never received has committed
nothing.

Two things from the original are deliberately absent: run-turn-execution.ts is
untouched, because its accounting distinguishes adapters that report their own
physical sends and carrying the PR's unconditional pre-count would double-charge
them; and no content HMAC exists anywhere here.

Also narrows the 400 refinement added earlier in this branch, after review: it now
consults only the LAST recovery recorded on the attempt, and a finalizer that can
prove a cause passes it directly instead. The key-account rotation now attributes
the attempt it seals, which previously reached the ledger with no attribution at all
because the finalization seam only ever sees the last attempt of a request.

Co-authored-by: yansigit <yansigit@users.noreply.github.com>
…contract

#5063 proposed retention on the canonical ledger, which is the right architecture:
the alternative is a projection that hides rows the ledger still has, and that is a
second retention policy. What its implementation could not promise is that a row
appended between its size snapshot and its rename survived -- it captured a size,
copied a suffix, and renamed over whatever was there. Its own concurrency test
performed two sequential calls and said so.

Two things close that here. The append is synchronous and the compaction runs inside
the same call stack, with no await between the append and the publication, so no
in-process append can interleave; a second server on the same home cannot append at
all, because it is refused by the existing ledger-owner lease at startup, which is
why the hook is installed after ownership rather than before. And
validateBeforeRename re-opens the target immediately before the rename and refuses
unless identity, size and revision metadata are byte-for-byte what was copied -- so
an append from anywhere else aborts the replacement rather than losing the row. Both
the original file and that append survive, and the next append retries from a fresh
revision. A test drives exactly that window through an injected hook, because a
contract nothing can drive is a contract nobody has checked.

Publication goes through the shared atomic writer rather than a hand-rolled temp
lifecycle, which is where the exclusive private temp, the identity assertions, the
platform-aware replace and the residual cleanup already live. The writer gains a
streaming form so the retained span is copied in bounded chunks instead of held in
memory as one string, and that form fsyncs the temp before the rename and does not
swallow the failure: a replacement whose replacement is not on disk can lose the
rows it was meant to keep.

Rows are copied byte for byte and never parsed or re-serialized. A retention pass
that understood the row shape would silently drop every field it was written before,
which for this branch would mean the failure stage and cause it just added.

The invalidation half was missing entirely from the original. Deleting rows
invalidates three readers that do not watch the file: the 2,000-entry Logs ring,
which otherwise keeps serving rows the ledger no longer has until eviction or a
restart; the retained usage aggregate and failure projection, whose checkpoints now
point past a boundary that moved; and the request-history index, whose source
identity changed. All three are discarded after a replacement.

This does NOT close #5063. The Usage-page control it also asks for is not here: this
branch may not build or run the GUI, so it cannot produce the screenshot that gate
requires, and shipping an unverifiable control is worse than shipping the policy the
control would set. The limit is settable in config.json today and the docs say so.

Co-authored-by: Vocllum <149675937+Vocllum@users.noreply.github.com>
Adversarial review of this branch found three cases where the derived cause was
wrong against real request paths rather than against the fabricated facts the
first test used.

A stream that dies mid-flight is reported as a SYNTHETIC 502 -- a tail this proxy
wrote, with transportPhase mid_stream and the attempt marked aborted. Read in status
order that 502 became upstream-fault, which claims the origin answered when it did
not. Transport evidence now outranks the numeric status. Both causes refuse an
automatic resend, so this is an accuracy fix rather than a safety one, but a label
an operator cannot trust is a label they stop reading.

402 had no branch and fell through to payload-rejected, which made quota-exhausted
unreachable and pointed an operator at the payload when the account is what has to
change.

transport-unsent was reachable only through a fabricated status 0: a real connect
failure is formatted as 502 by the dispatch path. Worse, it was the FALL-THROUGH,
and it is the one transport cause that permits an automatic resend. It is now
reachable only through causeHint, from a site that classified a pre-connect failure
and can prove it; everything else answers transport-ambiguous, which is the honest
classification for an unknown execution state and the safe direction for a
permission decision.

Review also found the streamed atomic replacement fsynced the temp's contents and
not the directory entry recording the rename, so a host losing power after a
successful call could leave the old ledger or an indeterminate directory. The
streaming form now syncs the parent directory. Only that form does: it is the one
making a durability claim, and charging every config write for a promise its callers
were never given is a different change.

The regression cases now use the production shapes -- a synthetic 502 after
mid_stream, an aborted stream, an upstream 502 that stays an upstream fault -- rather
than a status no transport produces.
…ating them

Three findings from the second adversarial review round.

A non-streaming turn delivers its whole answer as one body and calls no per-frame
recorder, so every buffered response persisted adapter events with zero relayed
ones. That is the adapter-to-client loss signal, raised on every buffered request,
which makes the signal worthless. The buffered seam now records its delivery from
the body it built: everything the adapter produced did reach the client, in one
piece, and the semantic bytes and side effects are read from the assembled output.
The body is read by field name rather than by the adapter event union, so a member
added later is not a merge-time exhaustiveness failure in a counter that does not
need one.

Two tests claimed their cross products came from the declared vocabularies and then
wrote the members out by hand, which is how an added member leaves an exhaustive
test green without being exercised. They now read REQUEST_TERMINAL_STATUSES,
REQUEST_CLOSE_REASONS and a transport-phase roster that is declared once in the
contract leaf and consumed by the ledger validator instead of being stated twice.

INV-RESEND-01 named two enforcing tests while the structure checker binds only the
first, so the second was prose-only assurance. The attribution rule is now its own
INV-ATTRIBUTION-01 with one binding, and the test names it so the binding is
readable from both sides.

Adds the lane record, including the two limits this branch does not close: the six
intermediate attempt finalizers that still reach the ledger unattributed, and the
successful-recovery case that can still misattribute a 400. It also records a
pre-existing defect found while reviewing the atomic writer -- its scrub fallback
opens with "wx" and so always fails on an existing temp -- which is left alone
because it predates this branch and sits on a security-adjacent path.
The file-size ratchet reported NEW_OVERSIZED on the first exact-head run.
src/server/request-log.ts carries the whole request-logging surface and was 1,962
lines against the repository's 2,000-line seed threshold; the attribution wiring
pushed it to 2,015.

The remedy is a move, never a number: the cap only ever goes down, and a threshold
is not something to negotiate with. The two places a stage and cause are decided and
written -- the finalization seam, and the attempt sealed by a key-account rotation --
now live in src/server/request-log-failure-attribution.ts. Behaviour is unchanged:
the same facts go in, the same attempt is stamped before the snapshot, and the same
pair reaches the row.

request-log.ts is 1,979 lines after the move. That is 21 lines of headroom, which
the lane record notes for whoever touches this file next.
Three exact-head failures, all from the new failure-cause counter and all in
assertions that counted by hand.

management-metrics-export.test.ts already derives its sample total from the closed
vocabularies -- its own comment says the literal "went stale the moment a bounded
label value was added, which is the failure mode this repository keeps hitting in
merges". The new counter's contribution is added to that arithmetic the same way.
Its HELP/TYPE assertion was the literal 7 the comment warns about, so it now reads
the metric names out of the exposition and asserts the two groups name the same set
exactly once each, which is what deterministic grouping means and what no added
metric can make stale.

The dashboard-union assertion matched the literal string "import type {
AttemptRecoveryKind", which broke when the import wrapped across lines to take the
three new names. It now matches the property it was testing -- the name arrives from
the contract leaf and the page declares no union of its own -- without depending on
how the import is formatted.

The public metrics table in the management-API reference gains the new series.
…wo of them

The streamed writer added a third openSync(path, "wx", 0o600) and the portability
test counted exactly two. The count was the weaker form of what it meant: the
property is that no temp writer in atomic-write.ts drops the O_CREAT bit, and that
holds for however many writers exist. It is now a set comparison over every
openSync on the temp path, which a fourth writer cannot make stale and an
unsafe spelling cannot pass.

The edit is line-neutral because that file sits exactly at its ratchet cap.
@lidge-jun
lidge-jun force-pushed the codex/260920-r5-telemetry-derived branch from cdbd802 to 714ddbf Compare September 20, 2026 15:23
@lidge-jun
lidge-jun marked this pull request as ready for review September 20, 2026 16:06
@github-actions
github-actions Bot marked this pull request as draft September 20, 2026 16:07
@lidge-jun
lidge-jun marked this pull request as ready for review September 20, 2026 16:08
@lidge-jun
lidge-jun merged commit 07e2ac9 into dev Sep 20, 2026
31 of 32 checks passed
@lidge-jun
lidge-jun deleted the codex/260920-r5-telemetry-derived branch September 20, 2026 16:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ea3974bb3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

...(signal ? { signal } : {}),
onEntry: entry => accumulator.add(entry),
});
retained = retain(accumulator, scan, false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Register the failure projection with the memory budget

When GET /api/usage?failures=1 scans a large ledger containing many distinct failure tuples, this assignment retains the entire group map indefinitely, but the new cache is absent from APP_OWNED_RETAINED_STORE_REGISTRATIONS and never calls enforceAppOwnedMemoryBudget() after publication. Consequently it can exceed appOwnedMemoryBudgetMb while remaining invisible to memory accounting and eviction; expose retained byte/age statistics, register the projection as an evictable cache, and enforce the budget after publishing it.

Useful? React with 👍 / 👎.

Comment on lines +187 to +189
if (url.searchParams.get("failures") === "1") {
const projection = await getFailureProjection();
return jsonResponse({

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the failure-projection query

This adds a public management-API mode whose response completely replaces the normal usage summary, but the edited management reference still documents only generic GET /api/usage and never mentions failures=1, its fields, or the scan cost. Operators therefore cannot discover or safely consume the new surface; add the query mode and response contract to docs-site/src/content/docs/reference/management-api.md.

AGENTS.md reference: AGENTS.md:L434-L435

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant