Skip to content

Derive macOS surface routing from one owner table (the crop classifier misreads frontmost-app and a missing surface) #2880

Description

@thymikee

Purpose

Bug class: a classification restated away from its owner. When a site keeps its own copy of a
rule, adding a new member to the domain, or changing the rule, updates some copies and misses
others. Examples fixed in the #2803 arc:

One instance remains, and it already disagrees: "which backend serves a macOS surface"
(line refs at origin/main 6debef0634):

Site Rule as written
packages/platform-apple/src/interactor.ts:212 (snapshot) isMacOs && surface && surface !== 'app' goes to the helper
packages/platform-apple/src/interactor.ts:432-437 usesMacOsSurfaceScreenshot (screenshot + --fullscreen refusal) same
packages/platform-apple/src/interactor.ts:440-442 usesMacOsHelperSurface (readTextAtPoint, :74) same
packages/platform-apple/src/interactions.ts:197 (press) same
packages/platform-apple/src/runtime-snapshot.ts:30-34 (runtime captureSnapshot goes to host.snapshot.captureSurface) same
packages/platform-apple/src/runtime-snapshot.ts:117 (native find admission refused) same
packages/platform-apple/src/os/macos/surface-snapshot.ts:11 (defensive TypeError) same, negated
packages/platform-apple/src/os/macos/helper.ts:375,378 runMacOsSnapshotAction same, as the type Exclude<SessionSurface, 'app'>
src/daemon/screenshot-crop-target.ts:101-103 classifyMacOsCropTarget app or frontmost-app gives macos-app-window, everything else including undefined gives macos-helper

The crop classifier gets two members wrong compared with the path that actually captures:

  • frontmost-app: the helper captures it. runAppleScreenshot goes to runMacOsScreenshotAction,
    and the helper always captures NSScreen.main (apple/macos-helper/.../main.swift:548-566).
    fix(macos): refuse --fullscreen on every helper-captured surface #2849's live validation confirmed this. The crop classifier still calls it an app-window frame.
  • undefined surface on a macOS session: every platform-apple route sends it to the runner, the
    same as app. The daemon also reads it as app elsewhere (session.surface ?? 'app' in
    session-close.ts:151, inventory.ts:123, application-lifecycle-recovery.ts:136). The crop
    classifier calls it macos-helper. src/daemon/__tests__/screenshot-crop-target.test.ts:34
    pins this wrong answer.

Nobody sees this today because both macOS crop cells are rejected pending pixel-identity
evidence (screenshot-crop-target.ts:48,50). The next planned step is to collect that evidence and
flip a cell to accepted. If macos-app-window is flipped using an app session, --crop-on on a
frontmost-app session will crop a main-display image with window-frame geometry, and nothing
will warn. The existing test at packages/platform-apple/src/__tests__/screenshot-macos-surface.test.ts:45
builds its expected domain from the implementation's own condition
(SESSION_SURFACES.filter((s) => s !== 'app')). That test is tautological.

A second gap: the helper entry points do not require a surface at all.
runMacOsReadTextAction (helper.ts:388-391), runMacOsPressAction (:429-434) and
runMacOsScreenshotAction (:474-476) take options.surface? with a = {} default, so any new
call can reach the helper without asking the routing rule.

Required behavior

  1. One owner, keyed by the domain. Put the table in packages/contracts/src/session-surface.ts,
    beside SESSION_SURFACES, and re-export the public names from
    packages/contracts/src/facades/session.ts (@agent-device/contracts/session):

    import type { SnapshotBackend } from '@agent-device/kernel/snapshot';
    
    /** The backend that serves every operation on a macOS surface. */
    export type MacOsSurfaceBackend = Extract<SnapshotBackend, 'xctest' | 'macos-helper'>;
    
    const MACOS_SURFACE_BACKENDS = {
      app: 'xctest',
      'frontmost-app': 'macos-helper',
      desktop: 'macos-helper',
      menubar: 'macos-helper',
    } as const satisfies Record<SessionSurface, MacOsSurfaceBackend>;
    
    /** An absent surface is an app session, the reading every route already gives it. */
    export function macOsSurfaceBackend(surface: SessionSurface | undefined): MacOsSurfaceBackend;
    
    type HelperRoutedSurface = {
      [S in SessionSurface]: (typeof MACOS_SURFACE_BACKENDS)[S] extends 'macos-helper' ? S : never;
    }[SessionSurface];
    
    declare const helperSurface: unique symbol;
    /** A surface the owner routed to the macOS helper; only `macOsHelperSurface` produces one. */
    export type MacOsHelperSurface = HelperRoutedSurface & { readonly [helperSurface]: true };
    export function macOsHelperSurface(surface: SessionSurface | undefined): MacOsHelperSurface | undefined;

    The brand's domain is derived from the table. Do not write it as Exclude<SessionSurface, 'app'>:
    that is one more copy of the rule, and it would accept a new surface that the table routes to
    'xctest'.

    Vocabulary. The values reuse the kernel SnapshotBackend channel names
    (packages/kernel/src/snapshot.ts:335-346) and do not add a third name for this split. The
    runner is the XCTest channel for every operation on an app surface (snapshot, screenshot
    through captureScreenshotViaRunner, press, readText, native find). The helper snapshot already
    reports backend: 'macos-helper'. The crop classifier's macos-helper / macos-app-window
    names a capture frame, which is a different thing. It stays daemon-private and maps from this
    value (item 2). The import is import type. contracts already depends on
    @agent-device/kernel/snapshot (snapshot-diagnostics.ts:1), so this adds no package edge and no
    eager module.

    The functions are device-agnostic. Callers keep the isMacOs(device) gate, because contracts
    must not value-import @agent-device/kernel/device into the session facade: that facade is
    a vocabulary entry, and the eager-closure budget allows it no growth. Record<SessionSurface, …>
    is what makes an unclassified new surface fail to compile.

  2. Every macOS routing site reads the owner. The platform-apple sites in the table above,
    and the crop classifier, derive their answer from macOsSurfaceBackend / macOsHelperSurface.
    The platform-apple sites narrow once and pass the branded value on, for example
    const helper = isMacOs(device) ? macOsHelperSurface(options.surface) : undefined; if (helper) ….
    Delete usesMacOsSurfaceScreenshot and usesMacOsHelperSurface. Keep the --fullscreen
    refusal in the helper branch, as fix(macos): refuse --fullscreen on every helper-captured surface #2849 has it.
    classifyMacOsCropTarget(surface) becomes
    macOsSurfaceBackend(surface) === 'macos-helper' ? 'macos-helper' : 'macos-app-window'.
    In runtime-snapshot.ts:109, tighten the local surface?: string in admitAppleNativeFind's
    input to SessionSurface, the type FindTextInput already has
    (packages/contracts/src/selector-observation-runtime.ts:6). Do not cast.

  3. The helper entry points require an owner-routed surface. Every helper entry point except
    runMacOsAlertAction has a required surface: MacOsHelperSurface and no = {}
    default
    on the options object that carries it. The type is a type-only import.

    • runMacOsSnapshotAction(surface: MacOsHelperSurface, options?) (helper.ts:374). Its
      result field surface (:378) is the helper's JSON echo, not an owner-routed value. Type it
      as SessionSurface.
    • runMacOsReadTextAction(x, y, options: { surface: MacOsHelperSurface; bundleId?: string })
      (:388).
    • runMacOsPressAction(x, y, options: { surface: MacOsHelperSurface; … }) (:429).
    • runMacOsScreenshotAction(outPath, options: { surface: MacOsHelperSurface }) (:474).
    • captureMacOsSurfaceSnapshot(options, signal) in os/macos/surface-snapshot.ts: the
      options type requires surface: MacOsHelperSurface. The runtime TypeError guard at :11
      is replaced by the parameter type. Its only unbranded caller, loadMacOsSurfaceSnapshot in
      src/platform-runtime-operation-host.ts:40-46, narrows through macOsHelperSurface and
      throws the same TypeError when the result is undefined. Because the options shape stays
      the same, src/__tests__/snapshot-desktop-surface.test.ts:72 does not change.

    appendMacOsHelperContextArgs keeps surface?: SessionSurface, because runMacOsAlertAction
    shares it. runMacOsAlertAction is out of scope: macOS alerts always go to the helper,
    whatever the surface.

    Do not cast to MacOsHelperSurface anywhere except inside the owner. Tests are not an
    exception: they get branded values through macOsHelperSurface('desktop')!.

  4. Tests assert behavior against an expectation written in the test, not derived from the owner.
    Each test declares its own Record<SessionSurface, MacOsSurfaceBackend> literal, plus an
    explicit undefined row. A new surface must then be classified in the owner and in each test
    before anything compiles.

  5. Loading shape stays the same.

    • helper.ts and surface-snapshot.ts are in the macos-facade.ts eager closure (about 24
      modules by an approximate static walk; the gate's count is authoritative). They gain only
      import type.
    • The new value imports of @agent-device/contracts/session go into interactor.ts,
      interactions.ts, runtime-snapshot.ts, src/daemon/screenshot-crop-target.ts and
      src/platform-runtime-operation-host.ts. None of these is in a facade or hub closure today
      (platform-apple index.ts loads them lazily).
    • interactor.ts already imports helper.ts (:7, runMacOsScreenshotAction) and
      surface-snapshot.ts (:32) statically. Keep those as they are. The dynamic
      await import('./os/macos/helper.ts') calls at interactor.ts:448, interactions.ts:226
      and surface-snapshot.ts:14 stay dynamic (ADR 0019 implementation laziness).

Completion conditions

  • src/daemon/__tests__/screenshot-crop-target.test.ts iterates SESSION_SURFACES plus undefined
    for a macOS device and checks each against a literal expectation map. The expected values are
    app and undefined give macos-app-window; frontmost-app, desktop and menubar give
    macos-helper. This fails on the old code for frontmost-app and undefined. Row :34
    moves to surface: 'desktop'.

  • The platform-apple interactor test (extend __tests__/screenshot-macos-surface.test.ts) iterates
    SESSION_SURFACES plus undefined on a macOS device, with the helper and runner entry points
    mocked. For each surface it asserts that screenshot, snapshot, readTextAtPoint and the
    primary single press all reach the backend named by the test's literal map. Four operations
    agreeing on the same answer is the property under test. The file may be renamed to
    interactor-macos-surface.test.ts as a pure rename. Line 45's
    SESSION_SURFACES.filter((s) => s !== 'app') is replaced by the literal map.

  • packages/platform-apple/src/runtime.test.ts (next to the test at :669) iterates the same
    domain. For each surface it asserts that runtime captureSnapshot calls
    host.snapshot.captureSurface exactly when the literal map says macos-helper, and that native
    find admission returns no runner route for those surfaces.

  • packages/platform-apple/src/os/macos/helper.test.ts passes branded values from
    macOsHelperSurface(…)! at every call that names a surface today: :37
    (runMacOsSnapshotAction('desktop')), :65-66, :107, :134, :165-166, :208 (press) and
    :229 (screenshot). No as never or other cast.

  • A typecheck guard in helper.test.ts fails CI if an entry point is widened back. It holds
    // @ts-expect-error over each of these calls:

    • a wrong literal: runMacOsSnapshotAction('desktop') and
      runMacOsScreenshotAction(out, { surface: 'menubar' });
    • an omitted surface: runMacOsReadTextAction(1, 2, { bundleId: 'com.example' }),
      runMacOsPressAction(1, 2, {}) and runMacOsScreenshotAction(out).

    If a parameter loses the brand, or the surface becomes optional again, the directive becomes
    unused and pnpm typecheck fails.

  • The PR description shows a local, uncommitted edit that adds a fifth member to
    SESSION_SURFACES. That edit must fail typecheck at MACOS_SURFACE_BACKENDS and at every test
    expectation map. Show the same edit classified as 'xctest' too: macOsHelperSurface must then
    refuse it, and a helper call with that surface must fail typecheck (the brand's domain comes from
    the table).

  • Supporting check only: git grep -nE "surface !== 'app'|surface === 'frontmost-app' \|\||Exclude<SessionSurface, 'app'>" -- packages/platform-apple/src src/daemon/screenshot-crop-target.ts src/platform-runtime-operation-host.ts ':!*test*'
    returns nothing. This grep does not catch list-style predicates such as
    ['desktop', 'menubar', 'frontmost-app'].includes(surface) or a flipped
    surface === 'app' ? … : …. The brand and the behavior tests above are the real guard.
    alert.ts:46 and open-policy.ts:42 answer different questions and stay as they are.

  • pnpm check:affected --run is green: typecheck, lint, layering (R9: no new package edge;
    contracts still does not import platform code), eager-closure-budgets (no growth for
    macos-facade.ts, src/cli.ts, src/platform-runtime.ts, or the session facade), fallow,
    and unit tests.

  • Docs and CHANGELOG: the crop refusal text for frontmost-app changes from macos-app-window to
    macos-helper, and for a surface-less macOS session it changes the other way. Code
    (UNSUPPORTED_OPERATION) and reason do not change. Add a one-line CHANGELOG entry only if
    the maintainer treats refusal text as user-visible.

Estimated cost

  • About one day for a Sonnet-class agent, in one PR.
  • Net production lines: about 0 to +15. The owner adds about 20 lines; deleting the two
    platform-apple predicates, the surface-snapshot.ts guard and the inline conditions removes
    about 15. The helper signatures change type but not length. This PR moves ownership and fixes a
    latent misclassification. It does not shrink the code.
  • Tests: about +120 to +170 lines, including the helper.test.ts call-site changes and the
    type guard. Gross diff: about 300 to 400 lines, well within the 1,000-line budget.
  • Smaller alternative: the table, macOsSurfaceBackend, the crop fix and the literal-map tests
    are about half the work and fix the latent bug. They do not stop a new helper call from skipping
    the owner. The brand (item 3) is what closes that, and it is worth its helper.test.ts churn
    only with the required surface and the table-derived domain above.

Non-goals

  • The daemon's "surface names no app" classification: src/daemon/handlers/session-state.ts:123-124,
    session-lifecycle/internal/inventory.ts:288-290,
    interaction/internal/interaction-touch-policy.ts:13, and
    src/commands/interaction/runtime/resolution.ts:763, all of which are desktop|menubar. This is a
    different question (frontmost-app falls on the other side), and all four copies agree today.
    Do not fold it into MacOsSurfaceBackend because the memberships happen to overlap. It can be a
    follow-up with its own owner.
  • Per-command open and alert target policies: platform-runtime-open-target.ts,
    platform-runtime-apple-application-tools.ts:165,194, alert.ts:46, open-policy.ts:42,
    surface-snapshot.ts:16 (menubar bundle). These are different questions.
  • Operations that ignore surface: fill, type, scroll, longpress, and gestures on a
    macOS frontmost-app session go to the runner, while its snapshot is read by the helper. This
    was seen in code only and has not been checked on a live session. It is out of scope; file it
    separately if a live check confirms a mismatch.
  • xctestrun disallowed hint lists (packages/platform-apple/src/runner/apple-runner-platform.ts:33-121).
    Each profile hand-lists the other profiles' SDK tokens, and the iOS, tvOS and macOS lists never
    gained xros/xrsimulator when visionOS was added. The test "stay unchanged when visionOS is
    added" from refactor: consolidate Apple platform internals #968 pins that. The drift is real but has no effect: scoreXctestrunCandidate
    only reorders candidates, findXctestrun returns the top one anyway, and derived dirs are
    per-platform unless AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH is shared. It is not worth a PR alone.
  • runnerSynthesizesTap (packages/kernel/src/device.ts:188). fix(ios): request runner synthesis only on platforms that synthesize #2835 already made it the one
    predicate for the Swift #if os(iOS) gate. Deriving it from resolveApplePlatformName would
    inherit that function's non-exhaustive default: 'iOS', which maps watchOS to iOS. That gains
    nothing.
  • TS and Swift runner command traits. Both are exhaustive already (TS: runner-command-traits.ts
    satisfies Record<…>; Swift: CommandType.traits switch). They differ on purpose (querySelector
    and uptime for resends). Do not merge them.
  • Renaming the crop targets. macos-helper / macos-app-window stay as the daemon's frame
    names; only their classification changes.
  • Changing which backend any surface uses, or accepting any crop cell.

Dependencies / related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions