fix(interaction): report an unsettled post-gesture surface instead of a definite miss

[thymikee]

Summary

Fixes the smoke:automation-input flake id="automation-longpress" did not become visible after scrolling (20 failures on 18 branches, 3 on main, and #2832). Post-scroll captures never agreed (post_gesture_snapshot_stabilization_timeout), so is visible returned a plain selector_not_found for a moving surface.

When the last poll pair disagrees, stabilization now stamps unsettledGesture on the tree. The tree is the only carrier. is/get/find/wait report it in error.details or data with a warning, through one helper shared with targetActivation (which now also lands on error.details). Mutating find keeps the captured tree. snapshot adds the warning in its existing builder. is absent refuses with observation: "unsettled", wait absent keeps polling, and the selector cache does not reuse the tree.

The E2E helper keeps the forward-only 0.75 × 3 search. On unsettledGesture or captureStalled it re-reads the same offset (at most twice per scroll) without using a scroll attempt. Follow-ups: #2853 (click/press/fill), #2856 (gestureNoEffect on the same stamp). The failing job's two Swift timeouts are fixed by #2847 and #2843. 22 files: production +134/−81, tests +220/−20.

Validation

Tested at c0b1b74541.

• pnpm check:affected --run: pass (6617 tests). Each new test fails under a named mutation.
• Live at 0c1c110373 (same behaviour; later commits only simplify it), iPhone 17 Pro / iOS 26.2, fling then read: find … click and is visible carry unsettledGesture in 6/6 trials; snapshot warns in 3/3. Main returned plain misses.
• smoke:automation-input: 2/5 on this head and 2/5 on main. Every failure is a pre-existing local get text id="automation-event-name" miss at bootstrap. The longpress search never failed.

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.79 MB	4.79 MB	+1.0 kB
Package (unpacked)	4.79 MB	4.79 MB	+1.0 kB
Package (download)	1.43 MB	1.43 MB	+396 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	20.8 ms	20.2 ms	-0.6 ms
CLI --help	55.5 ms	54.7 ms	-0.9 ms

[thymikee]

Reviewed at faec8d2. The test-side change looks sound, and there are no conflicts.

The failing Smoke Tests job fails in two Swift runner tests, testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption and testAlertAcceptDoesNotActivateAReplacementWithASharedButton. Both are timeouts. This PR only changes the TS integration test helpers, so these failures are likely unrelated.

Not blocking: the mock probe in test/integration/ios-simulator-e2e-visibility-scroll.test.ts#L78 ignores scroll position. With SCROLL_SEARCH_STEP at 0.4, the two up steps return to offsets 1.2 and 0.8, not to 0.4 where the test plants the stale read. Is the reversal meant to go back to that offset (for example with as many up steps as down steps and a position-aware mock), or are the extra probes simply retries? Forward reach also drops from 2.25 to 1.6 viewports for all three callers, so a target in that range would now fail.

[thymikee]

Live evidence for bb059ec (lp2 simulator, iPhone 17 Pro / iOS 26.2)

Unsettled path. Catalog tab: a fast swipe 200 760 200 160 fling, then an immediate is visible|absent id="catalog-footer". Six trials each on the same binary built from origin/main (9b76f60) and from this head.

Pre-fix (main): 3 post_gesture_snapshot_stabilization_timeout in the request logs, yet each miss was a plain selector_not_found and every is absent passed:

trial 1 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=null
trial 2 is absent: status=0 pass=true reason=- observation=- unsettledGesture=null
trial 3 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=null
trial 4 is absent: status=0 pass=true reason=- observation=- unsettledGesture=null
trial 5 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=null
trial 6 is absent: status=0 pass=true reason=- observation=- unsettledGesture=null

Post-fix: 6/6 reads hit the timeout with lastPairAgreed:false, and each response carries the typed fact:

trial 1 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=swipe 200 760 200 160
trial 2 is absent: status=1 pass=- reason=predicate_failed observation=unsettled unsettledGesture=swipe 200 760 200 160
trial 3 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=swipe 200 760 200 160
trial 4 is absent: status=1 pass=- reason=predicate_failed observation=unsettled unsettledGesture=swipe 200 760 200 160
trial 5 is visible: status=1 pass=- reason=selector_not_found observation=- unsettledGesture=swipe 200 760 200 160
trial 6 is absent: status=1 pass=- reason=predicate_failed observation=unsettled unsettledGesture=swipe 200 760 200 160

is absent response (trial 2):

{"success":false,"error":{"code":"COMMAND_FAILED","message":"is absent could not prove absence for selector id=\"catalog-footer\": the surface was still changing after swipe 200 760 200 160","details":{"command":"is","reason":"predicate_failed","predicate":"absent","selector":"id=\"catalog-footer\"","matches":0,"observation":"unsettled","unsettledGesture":{"action":"swipe","positionals":["200","760","200","160"]}}}}

The next read in each trial captured afresh, with no unsettledGesture field.

Scenario. smoke:automation-input five times in a row on this head: 5/5 PASS (60-88 s each). The unsettled path did not occur in these local runs, so the helper's re-probe was not exercised live; it is covered by the position-aware unit test.

Mutations (each makes a new test fail): no gestureUnsettled on the timeout outcome; no stamp on the tree; no selector disclosure; no unsettled absence observation (also wait absent); no selector-cache refusal; flag every timeout including a quiet rebased last pair; main's helper against the new helper tests.

[thymikee]

Both points are addressed in the rework at bb059ecf36. The PR is now a product fix, and the review thread above has the reasoning.

1. Position-aware mock. listWithUnsettledFirstReads in test/integration/ios-simulator-e2e-visibility-scroll.test.ts answers each probe from the simulated offset. Each scroll increments the offset, and the element is on screen only at offset 1. The first read after every scroll returns selector_not_found with unsettledGesture, which is the shape the daemon now produces. The test asserts probes [1, 2, 2] and scrolls [1]: the unsettled miss at offset 1 is re-probed at that same offset with no scroll, and the element is found. On main's helper this test fails (it scrolls past to offset 2 and runs out of attempts).
2. Forward reach. The bidirectional plan and the 0.4 step are gone. All three callers keep the original forward-only scroll down 0.75 × 3 (2.25 viewports), with no up leg. A re-probe on unsettledGesture does not use a scroll attempt. It is bounded to one per scroll window, the same way the captureStalled retries are bounded. A second test checks this bound when the element is really absent (7 probes, 3 scrolls, then failure with every step listed).

The two Swift timeouts are unrelated. They are the occupancy and alert flakes that #2847 and #2843 fix. Live evidence: #2839 (comment)

[thymikee]

Adversarial review of the earlier test-only head faec8d2 (the reasoning the rework follows):

• The CI artifact (run 35868158609) does not show a stale capture. In the attempt-2 is visible request, three AX-bridge captures taken 0.5–1.9 s after the scroll never matched, then post_gesture_snapshot_stabilization_timeout. The surface was still changing.
• packages/capture-kit/src/post-gesture-stability.ts returned the latest unsettled capture with only a warn diagnostic, and capturePostGestureStabilizedResult dropped the timeout. So is visible gave a definite selector_not_found for an element a correct scroll had brought on screen. Any agent gets the same false miss, so a test-only retry plan would have hidden a product defect.
• Invariant: a read after a gesture either evaluates a settled surface, or its response carries a typed unsettled signal, and absence verdicts never claim definite absence from an unsettled capture.
• The plan-shape unit test passed for any plan with an up step at position 5, so it did not reproduce the failure.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-24 05:48 UTC

[thymikee]

Follow-up review at d3f1988 (after faec8d2). The fix still holds: an unsettled post-gesture surface is reported as unsettled, not as a definite miss. The delta (field renames, the contracts disclosure helper, the tree-only carrier, and find provenance via captureProvenanceOf) keeps that behavior.

The live run was at bb059ec. The changes since then were checked by range-diff and the production-route unit tests, not by a new device run. Deferred interactions (click, press, fill by selector after a gesture) do not get the unsettled warning; the CHANGELOG already states this gap.

CI was still running at review time. Smoke Tests (smoke:automation-input) and Integration Tests exercise exactly this route, so their results matter for this PR.

[thymikee]

Rebased onto main at 0c1c110373 and simplified. Both blocking findings are fixed.

1. Mutating find: FindTargetTree is now CaptureProvenance & Pick<SnapshotState, …>, and the capture returns the captured tree itself, so there is no field-by-field copy left. find-unsettled-gesture-disclosure.test.ts fails on the old shape.
2. One carrier: only the kernel tree stamp (SnapshotState.unsettledGesture) remains. gesture-unsettled.ts, withPostGestureWarnings, the contracts module, and the annotation warning are deleted. snapshot derives its warning in buildSnapshotWarnings. The selector routes derive the typed field and warning in withCaptureDisclosures. There is one name (unsettledGesture), one type (PostGestureAction in kernel, also used by gestureNoEffect), and one join (describePostGestureAction).

Size, measured against the base: production went from +182/−41 to +100/−44, and tests from +489/−11 to +287/−10. There is one test per invariant, and each fails under a named mutation. check:affected --run passes. Selector-targeted interactions are follow-up #2853.

[thymikee]

Simplified at c0b1b74541 (rebased onto main). The changes: one tree-fact disclosure helper, shared with targetActivation, which now also lands on error.details. Both post-gesture wordings live in capture-kit/post-gesture-stability. The absence module owns one causes table plus isUnprovableAbsence. The session-cache predicate refuses an unsettled tree. The resolved capture stamps the tree, so readSnapshot only reads again. The E2E helper has one re-read budget. The find case is folded into its sibling test. Size: production +134/−81, tests +220/−20. Every test fails under a named mutation, and check:affected --run passes. Follow-ups: #2853 and #2856.

[thymikee]

c0b1b74 fixes what d3f1988 left open: the code now reports an unsettled post-gesture surface instead of a definite miss. This is ready for human merge. 20 checks pass on c0b1b74 and I don't see any conflicts. I read the tests only and didn't run the named mutations myself, and I reviewed the logical delta by diffing d3f1988 against c0b1b74 directly, since the packet's delta mixes in upstream rebase commits. Not blocking: withTargetActivationDisclosure now also puts targetActivation on error.details for failed click/press through https://github.com/callstack/agent-device/blob/c0b1b74/src/daemon/capture-disclosure.ts#L82, and since no test asserts that field on a failed response (the nearby test only checks it's absent with no capture), it'd be worth adding one for a repairing capture followed by a failed selector read, and updating the CHANGELOG to say "a failed response" — but this can be taken or left.