Skip to content

refactor(ios-runner): drop the runner's snapshot warning text; the verdict carries every disclosure - #2834

Merged
thymikee merged 2 commits into
refactor/2786-runner-typed-retry-codesfrom
refactor/2802-drop-swift-snapshot-warning-text
Sep 23, 2026
Merged

thymikee merged 2 commits into
refactor/2786-runner-typed-retry-codesfrom
refactor/2802-drop-swift-snapshot-warning-text

Conversation

@thymikee

@thymikee thymikee commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Summary

Stacked on #2831.

The runner stops writing its own snapshot warning text. legacyQualityMessage and customActionCoverageWarnings are deleted, along with their Swift tests. The stamped payload now forwards the underlying payload.message unchanged; in practice that is nil. The daemon already renders every one of these disclosures from snapshotQuality, and it ignores message whenever a verdict is present. The tree backend pin, the capability declarations, and the conformance coverage are unchanged. The host fallback for verdict-less payloads (interactor.ts) stays, because the issue scopes injected-provider compatibility out.

Tests:

  • A new TS runtime test covers a healthy capture that has partial custom-action coverage and a collapsed leaf, with no backend message. The warnings array comes from the verdict alone.
  • The provider test checks that a runner capture with no message keeps the verdict facts and adds no warnings of its own.
  • The Swift test testStampedPayloadCarriesDisclosuresOnlyInTheVerdict replaces the deleted message test, including its slot in the ios.yml PR list (final chore(gates) commit).

Where the issue differs from the code: the helpers were also used by tests in RunnerTests+AXSnapshotFallback.swift and RunnerTests+SnapshotPresentationInvariantTests.swift. The fixture app has no collapsed container, so the live check used a local Safari page with an aria-label tab strip.

6 files, +93/−143. Closes #2802.

Validation

Rebased onto main at a913cb222. Signed macOS host lane: 240/240 at 5be256801. check:affected --run is green at the stack head 60e533284. The evidence below is from before the rebase.

Tested at 40e80cb54.

  • pnpm check:affected --run passed. It fails open to the full set because ios.yml changed; fallow was included.
  • pnpm check:xctest-selection passed.
  • The signed macOS runner built, and the host lane ran 238/238 tests with 0 failures, including the changed ones.
  • Live check on the iPhone 17 Pro simulator, with each runner freshly built from its own source. The legacy strings appear 2× in the before binary and 0× in the after binary. The snapshot --json warnings arrays are byte-identical (cmp) for all four scenarios; they are listed below.
  • The forced-backend conformance passed for both tree and private-ax, on the before runner and on the after runner. It ran as a standalone driver because the full smoke E2E stops earlier, at snapshotDepthFrontier: the host Simulator AX bridge is circuit-disabled here, which is unrelated to this change.
  • On main, the iOS lane is already red on testAbandonedTreeCapture… and testAlertCannotProve….
  • Review: the Agent tool was unavailable, so I ran the code-review skill (high). 8 of its 9 findings are in lower stack layers. The one in this layer, "remove the host message fallback", was rejected because the issue says not to widen into injected-provider compatibility.
Before/after warnings arrays
  • Healthy capture with collapsed tabs (Safari page): before = after = ["Simulator AX snapshot unavailable (circuit-disabled); used XCTest for this app generation.","@e14 [Button] merges many labels into a single accessibility element. The app likely marks a container as accessible, …"], verdict {"state":"healthy","backend":"tree","collapsedLeafIndexes":[13]}
  • Capture limited by custom actions (fixture Catalog, --actions): before = after = ["Custom actions were read for 12 of 117 merged elements, on-screen ones first; the remaining 105 were not read, …"]
  • Recovered capture (deep tree 200x0, --actions, recovered/requested-backend): before = after = null
  • Recovered capture (conformance private-ax, recovered/deferred): before = after = ["Detected an overly complex or slow accessibility tree. Fell back to the private-ax snapshot backend. …"]
  • Capped tree (deep tree 200x0): before = after = ["This capture was cut at a backend limit, …"]

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.80 MB 4.79 MB -3.1 kB
Package (unpacked) 4.80 MB 4.79 MB -3.1 kB
Package (download) 1.43 MB 1.43 MB -962 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.3 ms 26.4 ms +0.2 ms
CLI --help 77.2 ms 77.5 ms +0.3 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 40e80cb. The runner no longer stamps a message, and TS already ignored message when a verdict is present, so users see the same warnings. The code looks right.

Not blocking: the new TS runtime and provider tests also pass on the old code, because TS already ignored message there. Only the Swift test testStampedPayloadCarriesDisclosuresOnlyInTheVerdict pins the deletion. Also, can any Swift producer emit a state outside {healthy, recovered, sparse}? I found no such path, but it is worth a check.

Smoke Tests was still queued at review time. The known failure at live-snapshot-depth-frontier.ts:126 also fails on main and #2832 fixes it.

This needs #2831 to merge first.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 23, 2026
@thymikee
thymikee added this pull request to stack #2830 September 23, 2026 17:20
thymikee and others added 2 commits September 23, 2026 19:48
…rdict carries every disclosure

The runner rendered recovery, sparse, custom-action coverage and depth warnings into the
snapshot payload's message for daemons that predate snapshotQuality. The daemon renders each
of those from the structured verdict and ignores message whenever a verdict is present, and
local runner adoption pairs only same-version runners, so the text had no reader. The stamped
payload now forwards the underlying payload message unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…unner PR lane

The deleted partial-coverage message test is replaced in the PR list by the test that pins the
stamped payload leaving disclosures to the verdict.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@thymikee
thymikee force-pushed the refactor/2802-drop-swift-snapshot-warning-text branch from 40e80cb to 5be2568 Compare September 23, 2026 18:08
@thymikee
thymikee merged commit dabbeb5 into main Sep 23, 2026
24 of 35 checks passed
@thymikee
thymikee deleted the refactor/2802-drop-swift-snapshot-warning-text branch September 23, 2026 18:13
@thymikee

Copy link
Copy Markdown
Member Author

Checked on main at 5c19230: no Swift producer can emit another state today, but the type does not prevent it.

  • SnapshotQuality.state is a plain String, not an enum (apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift:15). The stampedSnapshotPayload(... state: String ...) parameter is a plain string too (:680-683).
  • Every producer passes a literal: "recovered"/"healthy" (:394), "sparse" (:419, :423), and "sparse" from snapshotAccessibilityUnavailable (RunnerTests+Snapshot.swift:544-545). Outside UnitTests there is no other place that builds SnapshotQuality. Truncation also depends on state == "sparse" (:708).
  • On the TS side, readSnapshotQualityVerdict accepts only healthy, recovered and sparse (packages/capture-kit/src/snapshot-quality-verdict.ts:4-8). Any other value drops the whole verdict (:29-34). Since this PR removed the runner's warning text, a mistyped or one-sided new state would silently lose every disclosure. A second reader, packages/contracts/src/snapshot-capture-annotations.ts:91-97, casts any string state without checking it.

Filed #2872. It proposes a closed SnapshotQualityState enum, with the vocabulary pinned in a golden table under contracts/fixtures/ that both Swift and TS tests read, as ios-snapshot-backends.json already does for backends.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(ios-runner): delete Swift-side snapshot warning text; keep the tree pin

1 participant