Skip to content

Harden trusted runtime WAL recovery integrity#661

Merged
flyingrobots merged 3 commits into
mainfrom
fix/trusted-runtime-recovery-integrity
Jul 14, 2026
Merged

Harden trusted runtime WAL recovery integrity#661
flyingrobots merged 3 commits into
mainfrom
fix/trusted-runtime-recovery-integrity

Conversation

@flyingrobots

Copy link
Copy Markdown
Owner

Summary

  • cross-check recovered receipt-correlation parent sets against independently retained ingress envelopes before exposing read-only recovery
  • enforce exact per-transaction cardinality for singular submission and scheduler WAL authority frames
  • retain exact duplicates across separate transactions under the existing append-only/idempotency policy
  • correct the WAL topic's enumerated evidence count

Why

This is a post-merge integrity follow-up to #659. The final audit found that correlation-parent validation occurred during runtime restoration but not at the public TrustedRuntimeWal::recover_read_only boundary. It also found that the duplicate-frame correction covered tick receipts and correlations but left submission acceptance, retained-envelope, and state-delta cardinality inconsistent.

The result is one validation boundary for recovered causal-parent evidence and one exact-cardinality helper for every singular authority frame.

Validation

  • cargo +1.90.0 xtask pr-preflight --base origin/main
  • cargo test -p warp-core --lib --features native_rule_bootstrap,trusted_runtime trusted_runtime_host::tests:: -- --nocapture
  • cargo test -p warp-core --test external_consumer_contract_fixture_tests --features native_rule_bootstrap,trusted_runtime inverse_intent_resolves_one_admitted_transition_after_restart -- --exact --nocapture
  • cargo test -p warp-core --test trusted_runtime_host_loop_tests --features native_rule_bootstrap,trusted_runtime,host_test
  • cargo clippy -p warp-core --lib --features native_rule_bootstrap,trusted_runtime,host_test -- -D warnings -D missing_docs

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@flyingrobots, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e6c4d074-45f0-4e94-bf2f-09961f2eccae

📥 Commits

Reviewing files that changed from the base of the PR and between 4ef7db8 and 6d0571f.

📒 Files selected for processing (5)
  • CHANGELOG.md
  • crates/warp-core/src/coordinator.rs
  • crates/warp-core/src/head_inbox.rs
  • crates/warp-core/src/trusted_runtime_host.rs
  • docs/topics/WAL.md
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/trusted-runtime-recovery-integrity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Owner Author

Code Lawyer post-merge audit

Follow-up to #659. The merged branch was green, but the final audit identified the following remaining integrity defects:

# Severity Source Finding RED witness Commit Outcome
1 P1 Review follow-up TrustedRuntimeWal::recover_read_only exposed receipt correlations without comparing their causal-parent set to the independently retained ingress envelope Exact recovered-envelope/correlation mismatch failed because no validation boundary existed 4d9b5c77 Public recovery now rejects mismatched parent evidence with a typed error
2 P2 Self Submission acceptance, retained-envelope, and runtime state-delta frames lacked one consistent per-transaction cardinality law Structurally valid committed duplicate fixtures were first-selected or accepted 5c9d302e Every singular authority frame now uses one exact-cardinality helper
3 P5 Self The WAL topic announced four invariants while enumerating seven Exact documentation assertion failed 6d0571fa Count corrected

Local gate

  • full cargo +1.90.0 xtask pr-preflight --base origin/main
  • all 9 trusted-runtime WAL unit witnesses
  • all 32 trusted-host integration tests
  • real contract inverse admission/restart fixture
  • strict feature-gated Clippy with warnings and missing docs denied
  • Graft: no exported API or breaking changes

@codex review please

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
gh pr view 661 --repo flyingrobots/echo --json title,body,files,commits,additions,deletions,changedFiles > /tmp/pr.json
cat /tmp/pr.json | jq '.title, .additions, .deletions, .changedFiles'
jq -r '.files[].path' /tmp/pr.json

Length of output: 598


🏁 Script executed:

#!/bin/bash
set -euo pipefail
gh pr diff 661 --repo flyingrobots/echo > /tmp/pr.diff
wc -l /tmp/pr.diff
sed -n '1,400p' /tmp/pr.diff

Length of output: 18978


🏁 Script executed:

#!/bin/bash
set -euo pipefail
sed -n '400,499p' /tmp/pr.diff

Length of output: 4973


🏁 Script executed:

#!/bin/bash
set -euo pipefail
rg -n "witnessed_submissions" crates/warp-core/src/trusted_runtime_host.rs | head -50

Length of output: 1539


🏁 Script executed:

#!/bin/bash
set -euo pipefail
sed -n '200,270p' crates/warp-core/src/trusted_runtime_host.rs
echo ---
sed -n '1150,1200p' crates/warp-core/src/trusted_runtime_host.rs

Length of output: 5485


🏁 Script executed:

#!/bin/bash
set -euo pipefail
sed -n '1,90p' docs/topics/WAL.md | grep -n "^First\|^Second\|^Third\|^Fourth\|^Fifth\|^Sixth\|^Seventh\|^Eighth"

Length of output: 855


Confirmed: exactly seven items are enumerated, matching the corrected count. The implementation checks out. Everything lines up with the stated objectives.

@flyingrobots Reviewed. Findings match the diff:

  1. Correlation-parent validation at the public boundaryvalidate_recovered_causal_parent_evidence is called in TrustedRuntimeWal::recover_read_only (Line 1177 of crates/warp-core/src/trusted_runtime_host.rs) right before constructing TrustedRuntimeWalRecovery, comparing each ReceiptCorrelationPersistenceRecord.causal_parent_receipts against IngressEnvelope::canonical_causal_parent_receipt_refs() for the matching submission. Correlations whose submission has no retained envelope are skipped rather than erroring, which is correct since that's already the separate missing_submission_envelopes obstruction path, not a mismatch. ReceiptCorrelationCausalParentsMismatch is a clean, typed error carrying submission_id and the receipt's identity digest — good for diagnostics. Test recovered_correlation_rejects_parents_not_bound_by_envelope exercises the mismatch path directly with a tampered ticket_digest.

  2. Cardinality unificationunique_transaction_frame (returns Option) + required_unique_transaction_frame (returns Result) cleanly separate "at most one" (SubmissionEnvelopeRetained) from "exactly one" (SubmissionAcceptedRecorded, RuntimeStateDeltaRecorded, TickReceiptRecorded, ReceiptCorrelationRecorded). This closes the actual defect: submission_acceptance_record_from_transaction and the state-delta path in tick_records_from_transaction previously used .find()/first-match with no duplicate rejection — a structurally valid duplicate frame would have been silently first-selected instead of rejected. Now every singular authority frame goes through one helper. The new test runtime_wal_submission_record_selection_rejects_duplicate_singular_evidence and the extended runtime_wal_tick_record_selection_rejects_duplicate_singular_evidence (now also covering RuntimeStateDeltaRecorded) both assert WalDecodeError::InvalidEmbeddedFrame on duplicates. Per-transaction scoping is correct — nothing here penalizes exact duplicates that legitimately occur across separate transactions, consistent with append-only/idempotency.

  3. Doc count fixdocs/topics/WAL.md now enumerates First through Seventh and the text says "seven concrete things." Verified by count.

Minor observations, non-blocking:

  • The canonical_causal_parent_receipts free function moved into IngressEnvelope::canonical_causal_parent_receipt_refs() — good, this is exactly the kind of dedup that makes the new cross-check in trusted_runtime_host.rs trustworthy (same sort+dedup logic used for persistence and for the new validation, not two independently-maintained implementations that could drift).
  • validate_recovered_causal_parent_evidence builds a BTreeMap per call — fine at recovery-time cost, not a hot path.

No correctness issues found. LGTM.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Owner Author

Code Lawyer Activity Summary

Final reviewed head: 6d0571fa

# Severity Source File Commit Outcome
1 P1 Review follow-up crates/warp-core/src/trusted_runtime_host.rs 4d9b5c77 Read-only recovery cross-checks correlation parents against retained ingress evidence before projection
2 P2 Self crates/warp-core/src/trusted_runtime_host.rs 5c9d302e Singular intake and scheduler authority frames now have exact per-transaction cardinality
3 P5 Self docs/topics/WAL.md 6d0571fa Enumerated invariant count matches the seven documented items

Gate

  • CI: 35 successful, 0 pending, 0 failing
  • unresolved review threads: 0
  • changes requested: 0
  • CodeRabbit: explicit LGTM
  • local preflight, strict Clippy, restart fixture, and 32 trusted-host integration tests: green
  • worktree: clean

The repository still reports REVIEW_REQUIRED; the owner explicitly authorized an admin merge once CI and review state were clean.

@flyingrobots flyingrobots merged commit 4de34fc into main Jul 14, 2026
35 checks passed
@flyingrobots flyingrobots deleted the fix/trusted-runtime-recovery-integrity branch July 14, 2026 05:40

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d0571fa9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.find(|frame| frame.header.record_kind == WalRecordKind::SubmissionAcceptedRecorded)
.ok_or_else(missing_trusted_runtime_record)?;
let frame =
required_unique_transaction_frame(transaction, WalRecordKind::SubmissionAcceptedRecorded)?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject duplicate acceptance frames during read-only recovery

With a committed SubmissionIntake transaction that contains two identical SubmissionAcceptedRecorded frames, this new uniqueness check is not exercised by TrustedRuntimeWal::recover_read_only: that path builds submissions via recover_submission_index(&report), which scans every acceptance frame and treats identical repeats as idempotent. As a result, read-only recovery can still accept and expose a duplicate singular acceptance claim even though the new WAL invariant says duplicate singular authority frames must be rejected; enforce the same per-transaction uniqueness before or inside the recovered submission index path.

Useful? React with 👍 / 👎.

.iter()
.map(|record| (record.submission.submission_id, &record.envelope))
.collect::<BTreeMap<_, _>>();
for correlation in receipt_correlations {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate legacy correlation parents before exposing receipts

When a scheduler tick uses the legacy digest-only RuntimeStateDeltaRecorded payload, recover_runtime_state_delta_material records the state delta as missing and never emits a ReceiptCorrelationPersistenceRecord, but recover_read_only has already populated recovery.receipts from recover_receipt_index. Because this validation loop only checks receipt_correlations, a WAL can pair a retained envelope with mismatched correlation parents plus a legacy state delta and still expose the bad parent set through the public receipt index; validate the raw receipt-index correlations as well or reject before returning the recovery report.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant