test(ios-runner): pin production-built runner requests in a golden table TS and Swift both verify by thymikee · Pull Request #2900 · callstack/agent-device

thymikee · 2026-09-24T12:42:06Z

Summary

Adds contracts/fixtures/runner-requests.json, a golden table of production-built Apple runner requests (67 entries, 34 commands). TS producer tests drive every real call site (platform-apple entry points, both screen-recording modules) and assert an exact match; Swift tests decode every entry and require every CommandType case and every Command/SequenceStep/RunnerGesturePlan field to have a producer. Step 1 of #2881: proves a production site can build each request, not that the daemon sends it. Step 2 (Swift unit tests onto the table) is a separate follow-up PR.

Closes #2881 (step 1)

13 files touched. Net production lines: 0 TS, 0 Swift (one CaseIterable conformance on an existing line). Gross diff 894 lines (issue estimate 800-950; under the 1,000-line budget).

Validation

Tested at 08f33d6036.

pnpm check:affected --run: green (59 selected checks — format, lint, typecheck, layering, fallow, build, vitest, xctest-selection, packaged-runner-swift, and the rest). A tmpdir-leaks-model failure seen on one run traced to contention from other gates sharing the host (passes 7/7 alone); reran clean with the host idle.
pnpm check:xctest-selection: 322 declared methods, 0 unreachable by any lane.
Targeted vitest: runner/__tests__/runner-requests, __tests__/runner-requests, interactor-runner-provider, the new src/__tests__/screen-recording-runner-requests, and both root screen-recording tests — 6 files / 27 tests pass.
Swift: TEST BUILD SUCCEEDED (build:xcuitest:ios, unit tests included); 5/5 XCTest pass on a throwaway iOS 26.2 simulator (3 golden-table tests + 2 ErrorPayload wire tests).

Remaining risk: macOS host lane and iOS 18.x/tvOS runtimes for the new Swift tests were not run locally; CI-verifiable, not blockers.

github-actions · 2026-09-24T12:43:48Z

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.81 MB	4.81 MB	+14 B
Package (unpacked)	4.81 MB	4.81 MB	+14 B
Package (download)	1.44 MB	1.44 MB	+11 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.1 ms	27.4 ms	+0.3 ms
CLI --help	79.2 ms	81.0 ms	+1.7 ms

thymikee · 2026-09-24T13:39:10Z

Reviewed at e1f72f2. The golden table and the matching TS and Swift checks look right, and they follow the existing alert-command-traits pattern.

One question: src/__tests__/test-utils/runner-requests.ts repeats the fixture read and normalize logic from runner-requests-fixture.ts. Can the src tests use that helper, or does the package boundary require the copy?

Small notes, not blocking: the aborted-start assertion uses a bare .rejects.toThrow() that passes for any error (transport test#L576), and the new fixture exclusion in .oxfmtrc.json could use a one-line reason.

All checks pass, and there are no conflicts.

thymikee

Reviewed at e1f72f2. The golden table itself is the right direction. The remaining structural issue is that the guard and the root helper duplicate the fixture contract by hand, so they can drift from the table they are supposed to verify.

thymikee · 2026-09-24T15:22:25Z

+);
+
+test('runner request drives send only requests production builds', () => {
+  const sources = [


This anti-literal guard scans a hand-maintained source list, but the fixture already carries the real producer paths in entry.producer. It omits the root screen-recording producers, so a raw request literal or direct send in those files would not be checked. Derive the scanned sources from the fixture producers, or otherwise close the list against the producer set the fixture itself declares.

thymikee · 2026-09-24T15:22:25Z

+    fs.readFileSync(path.join(REPO_ROOT, 'contracts/fixtures/runner-requests.json'), 'utf8'),
+  ) as RunnerRequestEntry[];
+  const produced = captured
+    .map(([name, sent]) => ({ name, producer, request: JSON.parse(JSON.stringify(sent)) }))


This root helper copy has already diverged from the package helper: runner-requests-fixture.ts normalizes commandId and statusCommandId through wireRunnerRequest, while this compares the raw sent request. The current root entries happen to omit command IDs, but once a root producer carries one the comparison is against a different shape than the fixture. Use the shared normalization path if the package boundary allows, or move the normalization into a shared fixture utility.

…table Add contracts/fixtures/runner-requests.json: one entry per production request site, captured at its real entry point. Three producer tests own the entries: the platform-apple request sites, the runner-internal sites (real HTTP bodies from the fake runner), and the two recording modules. runner-contract.test.ts asserts every RunnerCommand has an entry, names are sorted and unique, and the drive code builds no request itself. Refs #2881

Every fixture request decodes as Command and re-encodes with the same key set, every CommandType case has a request, and every stored field of Command, SequenceStep and RunnerGesturePlan appears in some request. CommandType becomes CaseIterable for the case check. Refs #2881

Refs #2881

… tests Export the fixture reader as @agent-device/platform-apple/runner/requests-fixtures so root producers pin requests through the same commandId normalization, and derive the request-literal guard's sources from the producers the fixture declares. Refs #2881

…lden file The request-literal guard scans every fixture producer, so root behavior tests that doubled as producers had to weaken their exact request assertions. Move the root recording drives into src/__tests__/screen-recording-runner-requests.test.ts, restore the original literals, and pin the recordStop sent without an app bundle. Producers must now be *runner-requests.test.ts files that call assertProducedRunnerRequests(import.meta.filename, ...), so a comment no longer satisfies the check. The runner-internal drive moves to runner/__tests__/runner-requests.test.ts to follow that rule. Refs #2881

thymikee · 2026-09-24T18:10:11Z

Addressed the review at e1f72f2. New head 08f33d6.

Reuse point (root helper duplicating fixture read/normalize): src/__tests__/test-utils/runner-requests.ts is deleted. The fixture reader and its wireRunnerRequest commandId/statusCommandId normalization now live in one place, packages/platform-apple/src/runner/runner-requests.fixtures.ts, exported as the test-only subpath @agent-device/platform-apple/runner/requests-fixtures (same pattern as the existing host-kit/selectors -fixtures subpaths). Root producers import assertProducedRunnerRequests from it, so both sides go through the same normalization path.

Inline: guard scans a hand-kept source list: the request-literal guard now derives its scanned sources from readRunnerRequestFixture()'s producer field instead of a hand-maintained list, so it can't silently drop a producer again.

Inline: root helper skipped commandId normalization: closed by the same fixture-module move above — there's no longer a second normalization path to drift.

Follow-up finding while widening the guard: deriving sources from producers pulled the two root behavior tests (which doubled as producers) into the guard, which had weakened three request assertions to expect.anything() to pass it — a real loss of precision the review's shape was steering toward but hadn't flagged yet. Fixed by splitting concerns: a new dedicated drive, src/__tests__/screen-recording-runner-requests.test.ts, does the golden-table producing; the two root behavior tests are restored byte-for-byte to main (full request literals back, including recordStop with no appBundleId, now pinned as its own fixture entry). The producer-set rule was tightened to require both a *runner-requests.test.ts filename and a call to assertProducedRunnerRequests(import.meta.filename, ...), closing the gap where a bare comment satisfied the old check.

Rebase: onto origin/main (fa1082d); no conflicts. Confirmed no CHANGELOG.md edits (already removed on main, #2922; this branch never touched it).

Declined: fencing the new -fixtures subpath to test-only importers — none of the existing -fixtures subpaths have that fence, so a one-off rule for this one isn't warranted (a general fence is a separate layering change). Also declined merging the Swift decode helper with the general fixture-URL helper across 8 unrelated test files — out of scope here.

Gate on 08f33d6: pnpm check:affected --run green (59 checks: format, lint, typecheck, layering, fallow, build, vitest, xctest-selection, packaged-runner-swift, …). check:xctest-selection: 322 methods, 0 unreachable. Targeted vitest: 6 files / 27 tests pass. Swift: test build succeeded, 5/5 XCTest pass on a throwaway iOS 26.2 simulator.

thymikee · 2026-09-24T18:39:18Z

The code at 08f33d6 looks good. The evidence gap from the earlier pass (e1f72f2) is closed: the golden table is now pinned and verified on both the TS and Swift sides.

CI failure is unrelated to this diff. The PR's logical change does not touch .github; the CI config diffs in the stat are rebase content from #2875 and #2896. In job 107767200188, "Restore and build iOS XCTest runner" reached "** TEST BUILD SUCCEEDED **", including the Swift ModelsTests change here compiling cleanly, then failed on the toolchain probe (xcrun --sdk iphonesimulator --show-sdk-version ETIMEDOUT), the known cold-toolchain flake. The "selected but skipped" assertion only follows from that failed probe step.

Re-run Smoke Tests to clear the cold xcrun timeout; no code change is needed here.

I did not re-run vitest, check:affected, or the Swift XCTest suite locally, so the author's gate report is unverified, and I can't confirm the ETIMEDOUT is transient without seeing a re-run.

github-actions · 2026-09-24T18:44:01Z

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-24 18:43 UTC

thymikee commented Sep 24, 2026

View reviewed changes

thymikee added 6 commits September 24, 2026 19:46

chore(gates): keep runner-requests.json entries on one line each

58906cd

Refs #2881

test(apple-runner): pin the macOS desktopScroll pixels request

bbdf442

thymikee force-pushed the test/2881-runner-request-golden branch from e1f72f2 to 08f33d6 Compare September 24, 2026 18:09

thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 24, 2026

thymikee merged commit 6428c54 into main Sep 24, 2026
18 of 19 checks passed

thymikee deleted the test/2881-runner-request-golden branch September 24, 2026 18:42

okwasniewski mentioned this pull request Sep 24, 2026

feat(ios): appstate reads the session app's XCUIApplication state through the runner #2929

Merged

This was referenced Sep 24, 2026

fix(ios-runner): give app launch its own policy axis #2899

Merged

refactor(ios-runner): make off-main access to main-owned capture state fail the runner gates #2911

Merged

Navigation Menu

Uh oh!

test(ios-runner): pin production-built runner requests in a golden table TS and Swift both verify - #2900

test(ios-runner): pin production-built runner requests in a golden table TS and Swift both verify#2900
thymikee merged 6 commits into
mainfrom
test/2881-runner-request-golden

thymikee commented Sep 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 24, 2026 •

edited

Loading

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

thymikee left a comment

Uh oh!

thymikee Sep 24, 2026

Uh oh!

thymikee Sep 24, 2026

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

thymikee commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

github-actions Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size Report

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

thymikee left a comment

Choose a reason for hiding this comment

Uh oh!

thymikee Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

thymikee Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

thymikee commented Sep 24, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

thymikee commented Sep 24, 2026 •

edited

Loading

github-actions Bot commented Sep 24, 2026 •

edited

Loading