Skip to content

Add SwiftUI support - #41

Merged
daisuke0131 merged 23 commits into
masterfrom
feat/swiftui-support
Aug 9, 2026
Merged

Add SwiftUI support#41
daisuke0131 merged 23 commits into
masterfrom
feat/swiftui-support

Conversation

@daisuke0131

@daisuke0131 daisuke0131 commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • Detect SwiftUI Text / Image / Button through the accessibility elements SwiftUI publishes: new MeasurementTarget abstraction, AccessibilityElementScanner, and overlay integration with cross-framework (UIKit x SwiftUI) distance measurement
  • Detect List rows: List inserts a UICollectionView between the hosting view and row content, so the scanner reads accessibility elements from every view in a hosting subtree (rows measure one element per row, matching how List publishes them)
  • Block touches on the app while measuring is ON: a transparent shield absorbs app-bound taps, scrolls, and edge swipes, so accidental interaction can't fire actions or trigger a screen transition that would discard the measurement overlay. Only ViewMonitor's own UI receives touches; toggle OFF to interact with the app
  • All monitor buttons (UIKit targets and SwiftUI elements) attach to the root view above the shield, in window coordinates. Effectively invisible targets (hidden or alpha 0) get no button
  • Add View.viewMonitor() so apps using the SwiftUI lifecycle can start the monitor without a SceneDelegate
  • Add ViewMonitor.enableSwiftUIElementDetection(): iOS builds the accessibility tree only while an accessibility client is attached, so without this call SwiftUI detection finds nothing in a normal run. It starts tree construction in-process the same way UI test runners do. Debug builds only — the implementation is compiled out of release builds (no-op returning false, no private-API symbol strings in the binary)
  • When SwiftUI content is present but no accessibility elements can be detected, the info panel now shows setup instructions instead of failing silently
  • Render InfoView rows with SwiftUI internally (UIKit contract unchanged)
  • Add a SwiftUI lifecycle example app (Example/ViewMonitorSwiftUIExample, generated by a Ruby script like the existing example) and build it on CI

Known limitations

  • Monitor buttons are fixed at window coordinates and do not follow scrolling; with the input shield, scrolling while measuring is blocked anyway — toggle OFF, scroll, toggle ON to re-scan
  • .accessibilityElement(children: .combine) regions and List rows measure as one element; .accessibilityHidden(true) views are not detected; font/background/cornerRadius show None

Notes for reviewers

  • Manual testing on device surfaced four significant issues, all fixed here:
    • InfoView's internal UIHostingController fires viewDidAppear when the overlay is shown; the swizzled transition hook mistook that for an app screen transition and reload() tore down the overlay that had just been presented — the toggle appeared to never turn on
    • The previously documented simulator workaround (defaults write com.apple.Accessibility AutomationEnabled 1) does not actually work — the tree stays empty even after a simulator reboot. Replaced by enableSwiftUIElementDetection()
    • List rows were not detected at all, because row elements are published by CellHostingView (not _UIHostingView-prefixed) inside collection view cells
    • With monitor buttons inside their target views, taps on them died under the shield: passing them through requires re-running the hit test through _UIHostingView.hitTest, which is not a pure function — on device it returned the monitor button on the first call and itself on the second for the same point (captured via on-device logging). Buttons now attach above the shield instead, so no touch ever routes through SwiftUI's hit testing
  • An earlier version of this branch deleted a unit test against a real UIHostingController accessibility tree because the tree never materialized in the unit-test host. In-process activation unblocks it: SwiftUIElementDetectionTests now verifies Text/Image/Button detection and List row detection against real hosting views rather than fakes
  • Earlier launcher fixes retained from the original review round: z-order fix for the launcher vs element buttons, and a launcher retain-cycle fix with a deallocation regression test

Test plan

  • 122 tests / 20 suites green; swiftlint --strict 0 violations; both example apps build
  • Unit tests cover: injected fake accessibility elements (scanner, inspector, overlay, distance, z-order, deallocation), hosting-subtree scanning incl. dedup and the pure-UIKit guard, real UIHostingController trees end-to-end (VStack content and List rows), the InfoView-triggered teardown regression, the "no elements detected" notice, and the input shield (absorb, button placement above the shield, invisible-target skip, z-order, removal on hide)
  • Verified on a clean simulator (no external accessibility client): toggle ON shows measurement buttons over all UIKit and SwiftUI elements in both example apps — including every visible List row — with hit-testing resolving to the shield on empty areas and to measurement buttons over targets
  • Verified on device: toggle, UIKit-label measurement, and SwiftUI element measurement (on-device touch logging traced the fix for the shield/hit-test issue)

🤖 Generated with Claude Code

daisuke0131 and others added 23 commits August 6, 2026 22:51
SwiftUI accessibility elements cannot provide alpha or cornerRadius and
carry their content as accessibilityLabel, so ViewInspection needs
optional fields and a conditional text row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SwiftUI's Text/Image/Button never create UIKit views, so the scanner
walks the accessibility tree that SwiftUI publishes for VoiceOver
(public API only) and classifies elements by their traits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SwiftUI targets have no backing UIView, so their monitor buttons attach
to the root view at the element's window frame (no scroll tracking) and
inspections re-read accessibilityFrame at selection time.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Accessibility-element buttons attach directly to rootView after
infoView, so they painted and hit-tested above it whenever their frame
overlapped the info panel — obscuring it and stealing its drag
gesture. Re-assert bringSubviewToFront(infoView) after attaching all
target buttons in show(on:).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SwiftUI apps have no SceneDelegate, so expose a root-view modifier that
starts the monitor on first appear. Screen-change detection keeps using
the existing viewDidAppear swizzling, which UIHostingController hits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
InfoView keeps its UIKit contract (frame-managed size, drag by the
overlay) and delegates drawing to a hosted InfoRowsView.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MonitorOverlay adds SwiftUI accessibility-element monitor buttons
directly to rootView, landing them above the launcher button that
ViewMonitor.reload() had already added. If a SwiftUI element overlaps
the launcher's top-right corner, taps meant to stop measurement hit
the element button instead and the launcher's drag gesture is blocked.

Re-front the launcher after overlay.show(on:) in the onToggle closure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Example/ViewMonitorSwiftUIExample was added without being listed in
.swiftlint.yml's included paths, leaving it unlinted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The removed targetView property no longer exists; the test now
exercises measurementTarget, so its display name and function name
should say so.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
button.onToggle captured its own button strongly while removeLauncherButton()
never clears onToggle, forming a button -> onToggle -> button self-cycle.
Since reload() runs on every viewDidAppear/orientation change while started,
every screen transition leaked the superseded launcher instance.

Capture button weakly alongside self, matching the existing weak-self pattern.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
InfoView now renders through a UIHostingController, whose viewDidAppear
fires when the overlay is shown. The swizzled hook mistook it for an app
screen transition and called reload(), which tore down the overlay that
had just been presented and replaced the launcher with a fresh OFF
instance - on device the toggle appeared to never turn on.

Mark ViewMonitor's internal hosting controller with
MonitorInternalViewController and skip transition detection for it.
Excluding UIHostingController as a whole would also swallow real
transitions in SwiftUI apps, so only ViewMonitor's own controllers are
excluded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
iOS builds the accessibility tree only while an accessibility client is
attached, so SwiftUI detection silently found nothing in normal runs.
The README workaround (simctl defaults write AutomationEnabled) turned
out not to work at all: the tree stays empty even after a simulator
reboot with the key set.

Add ViewMonitor.enableSwiftUIElementDetection(), which starts tree
construction in-process the same way UI test runners do
(_AXSSetAutomationEnabled). The implementation is compiled only into
DEBUG builds; release builds get a no-op returning false and contain no
private-API symbol strings. The SwiftUI example app now calls it at
startup, and measurement buttons appear over all SwiftUI elements on a
clean simulator with no external tooling.

This also unblocks the end-to-end test that was written and deleted
earlier in this branch because the tree never materialized in the
unit-test host: with in-process activation it materializes immediately,
so the scanner is now covered against a real UIHostingController tree
instead of only fakes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the scanner finds a hosting view but zero accessibility elements,
the overlay used to show nothing at all, which is indistinguishable from
a bug (and was reported as one). Show instructions in the InfoView
instead: call enableSwiftUIElementDetection() or attach an accessibility
client. Pure UIKit screens are unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The documented simctl defaults write AutomationEnabled workaround does
not actually build the accessibility tree (verified on iOS 26 simulator,
including with a reboot). Document enableSwiftUIElementDetection() as
the primary path, keep Accessibility Inspector / VoiceOver as the
alternative, and cover both example apps in the device signing setup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SwiftUI's List inserts a UICollectionView between the hosting view and
the row content, and each row's accessibility elements are published by
a CellHostingView inside the cell - a class that does not match the
_UIHostingView prefix. Reading only the hosting view's own
accessibilityElements therefore found no rows at all: the hosting
view's array contains just the collection view (a UIView, skipped to
avoid double detection, with no AX children of its own to recurse
into).

Once a hosting view is entered, read accessibility elements from every
view in its subtree instead. Pure UIKit hierarchies keep the previous
behavior (no accessibility scan), and elements published by more than
one view are deduplicated by identity.

Covered by a structural unit test mirroring the List layout and an
end-to-end test against a real List in a UIHostingController; rows are
measured one element per row, matching how List publishes them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Taps on areas not covered by a measurement button used to reach the app:
actions fired, lists scrolled (leaving SwiftUI element buttons at stale
positions), and any resulting screen transition made the reload teardown
discard the overlay and reset the toggle to off.

Insert a transparent shield between the app and the measurement UI while
the overlay is shown. It absorbs app-bound touches; the info panel,
SwiftUI element buttons, and the launcher sit above it, and touches
aimed at UIKit-target measurement buttons (which live inside the app
hierarchy, below the shield) are let through via hit-test requery with a
re-entrancy guard. Toggling isHidden during hitTest is not an option:
some internal UIKit hit-test paths re-enter the override without
checking hidden, overflowing the stack (verified via crash report).

To interact with the app (e.g. scroll further down a list), toggle off,
move the screen, and toggle on again to re-scan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e targets

On device, tapping a UIKit-target monitor button (nav bar labels) did
nothing while measuring. The shield passed such touches through by
re-running the hit test, but the requery traverses
_UIHostingView.hitTest, which is not a pure function: for the same
point it returned the monitor button on the first call and itself on
the second (captured via on-device logging), so the final hit went to
the shield and the button never received the touch.

Stop relying on hit-testing through SwiftUI entirely: attach UIKit
target buttons to the rootView above the shield, the same placement
SwiftUI element buttons already use, with frames converted to window
coordinates. The shield becomes a plain absorb-everything view (the
requery and its re-entrancy guard are gone), and forcing
isUserInteractionEnabled on target views is no longer needed.

Since buttons are now fixed-position, targets that are effectively
invisible (hidden or alpha 0, e.g. the inline nav title label while the
large title is shown) get no button - previously their buttons were
invisible together with the hidden parent, but at window level they
would float over nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Monitor buttons are now fixed at window coordinates for UIKit targets
too, so the scroll-following note no longer belongs to the SwiftUI
limitations list. Fold it into the general measuring section, and
correct the transition behavior: the overlay closes and the toggle
returns to OFF, it does not refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@daisuke0131
daisuke0131 merged commit 0168c2f into master Aug 9, 2026
4 checks passed
@daisuke0131
daisuke0131 deleted the feat/swiftui-support branch August 9, 2026 05:56
@daisuke0131 daisuke0131 mentioned this pull request Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant