e2e: harden Android + iOS flows against system-UI flakes - #22
Open
tiensonqin wants to merge 13 commits into
Open
tiensonqin wants to merge 13 commits into
tiensonqin wants to merge 13 commits into
Conversation
Two consecutive CI runs failed the very first assertion (button.hosted-sign-in, 30s timeout) with a healthy app rendered behind a 'Pixel Launcher isn't responding' dialog — a system ANR occludes the entire a11y tree so Maestro sees only the dialog window. Start a background watchdog that polls 'uiautomator dump' every 3s and taps the ANR dialog's Wait button; skip with LOGSEQ_CHAT_ANDROID_E2E_SKIP_ANR_WATCHDOG=1. The runner self-test stubs an ANR dump and asserts the Wait tap lands.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
A PR run failed the setup flow after ~98s in 'when: Enter your username': each hierarchy query against the Cognito hosted UI (system browser) was stalling 15-30s+ on the loaded runner, then the XCTest driver threw an unhandled assertion mid-snapshot and Maestro exited 1. Give the condition real headroom via a hoisted optional extendedWaitUntil (120s) and retry the setup flow once so a driver crash does not fail the suite; LOGSEQ_CHAT_IOS_E2E_SETUP_RETRIES=0 opts out.
devin-ai-integration
Bot
changed the title
Each Safari WebView hierarchy fetch took ~93s on the loaded runner before returning, so a 120s budget only covered about one snapshot round. 240s gives the condition 2-3 real evaluations.
The watchdog fired every ~11s for 4+ minutes but the Pixel Launcher ANR kept re-triggering: 'Wait' only postpones the dialog while the process stays hung. Tap 'Close app' instead so Android restarts the hung process (fall back to 'Wait' when absent), suppress background ANR dialogs via 'settings put global anr_show_background 0', and raise AVD RAM to 4G — lowmemorykiller stalls are a plausible trigger under Gradle + emulator CPU contention. Self-test updated to stub and assert the Close app tap.
emulator-options made the action pick the latest emulator (37.1.11), whose qemu binary fails to load on ubuntu-latest (libpulse.so.0 missing) — the emulator never booted. Keep the Close-app watchdog and anr_show_background settings, which address the observed dialog directly.
android-signed-out went green for the first time in CI, then the next flow died before starting: the Maestro Android driver failed to come up within its 15s default on a loaded emulator. Export MAESTRO_DRIVER_STARTUP_TIMEOUT=180000 and retry each flow once (LOGSEQ_CHAT_ANDROID_E2E_RETRIES=0 opts out).
Swift print() never reaches the captured device-simulator.log, so the previous DEBUG-only apply traces were invisible in CI. os_log output lands in the harness log: each applied patch generation, stale-epoch drops (silent wedge suspect), and apply failures.
info-level Logger entries are filtered out of the captured simulator log — promote the apply trace to notice, stale-epoch drops to warning (already error-level), and always log effect-executor failures and deduped core responses (both previously invisible).
The else-branch fprintf(stderr) never reaches the captured simulator log, so an OCaml exception inside resolveEffect/applySnapshot on iOS is invisible — matching the silent sheet-stuck wedge. os_log_error lands in the unified log that Maestro's debug artifacts capture.
The loaded emulator took >180s to install and start the Maestro driver this run; the retry then raced the same slow startup.
logger.info already bracketed every performAsyncAndWait with 'Core action started/returned', but os_log info is dropped from the captured device-simulator.log (only notice+ survives). Mirror to notice so the stalled action is directly visible, and log the drain's effect start/resolved/startSync boundaries to pin the exact await that wedges post create-graph.
The iOS wedge leaves the drain stalled with zero diagnostics — the last visible event is 'effect resolved' followed by silence, so the hang is either inside an OCaml call on the executor thread or in caml_acquire_runtime_system itself. Logging at all three boundaries distinguishes them in the next run's device-simulator.log: 'enter' without 'acquired' means the OCaml domain lock was never released by a previous call; 'acquired' without 'done' means the hang is inside the named OCaml entry point.
…epro Signal.stabilize has no round cap — a task that unconditionally re-dirties the scheduler loops forever inside Lui_app.flush, the leading hypothesis for the iOS e2e wedge (executor dead inside a native.* FFI call after 'effect resolved'). Repin ocaml-signal to devin/stabilize-round-cap (logseq/ocaml-signal@b9cb81f) which raises Stabilization_limit_exceeded instead — the exception crosses the FFI boundary as lui_exception, surfacing the wedge in logs instead of a silent hang. The headless repro drives the sign-in resolve + snapshot flush sequence through the real app (green locally — the loop, if it exists, is in a different flush).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two infrastructure-level flakes hit consecutive e2e runs; the app itself was healthy in both.
Android — system ANR dialog occludes the a11y tree
Two runs failed the very first assertion —
button.hosted-sign-in(30s timeout) — while the app rendered correctly behind a "Pixel Launcher isn't responding" dialog. A system ANR takes the foreground window and hides the entire app a11y tree, so Maestro sees only the dialog.test-android-e2e.shnow starts a background watchdog after device detection: every 3s it pollsadb exec-out uiautomator dump /dev/tty; when the dump containsisn't respondingit extracts theWaitbutton's bounds and taps its center. Killed by the EXIT trap;LOGSEQ_CHAT_ANDROID_E2E_SKIP_ANR_WATCHDOG=1opts out. The runner self-test stubs an ANR dump and asserts the Wait tap lands — verified locally.iOS — XCTest driver stalls/crashes on the Cognito system browser
A run failed the setup flow after ~98s inside
when: Enter your username: each hierarchy query against the Cognito hosted UI (ASWebAuthenticationSession) was stalling 15–30s+ on the loaded runner (SpringBoard responsiveness timeouts in the sim log), and the driver then threw an unhandled assertion mid-snapshot — Maestro exited 1.ios-local-graph-setup.yamlnow gives that condition real headroom via a hoisted optionalextendedWaitUntil(120s), andtest-ios-e2e.shretries the setup flow once viarun_setup_flowso a driver crash doesn't fail the suite;LOGSEQ_CHAT_IOS_E2E_SETUP_RETRIES=0opts out.Link to Devin session: https://app.devin.ai/sessions/9a91e201984a4d4f96ce3e0f86ac6668
Open in Devin Desktop: https://app.devin.ai/desktop/session/9a91e201984a4d4f96ce3e0f86ac6668?variant=devin
Requested by: @tiensonqin