fix(a11y): keep the binary flow alive on a degenerate config and stop double scanning - #186
Open
kamal-kaur04 wants to merge 2 commits into
Open
fix(a11y): keep the binary flow alive on a degenerate config and stop double scanning#186kamal-kaur04 wants to merge 2 commits into
kamal-kaur04 wants to merge 2 commits into
Conversation
… double scanning
Two defects that only surface together, and only away from production hostnames.
1. loadModules() dereferenced config.apis unconditionally. A config with no apis
block — an auth failure, or a config server that never answered (measured: a
60s hang, after which the binary echoes the input config back) — threw here,
the caller tore the binary down, and the entire run silently fell through to
the Direct flow. Keeping the default endpoints is strictly better than losing
the binary flow.
2. getCloudProvider() classifies a session by hostname.includes('browserstack'),
so a hub served from a host without that substring reads as a third-party
grid. isBrowserstackSession() then goes false, _isCliAccessibilityFlow() with
it, and the service runs the classic accessibility handler WHILE the binary
runs its CLI accessibility module — both wrapping the same commands. Measured
on one such run: 22 classic + 19 CLI scans where 19 were expected. Duplicate
scans inflate scan_count against effective_expected and corrupt the App-A11y
stability metric.
The gate asks the CLI module registry rather than the hostname, and asks at
SCAN time rather than session start: service.ts must choose a flow before the
binary has necessarily finished booting, so a decision made then can be stale
by the time a command fires.
Verified: 48/48 accessibility-handler tests pass. service + cli suites show 4
failures, identical to untouched main (cliUtils network tests).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this about?
Two defects that only surface together, and only away from production hostnames. Both were found while verifying SDK-7422 on an internal environment; neither is specific to that environment, and the second one corrupts a production metric.
1. A config with no
apisblock killed the binary flow entirely.loadModules()dereferencedthis.config.apisunconditionally. A degenerate config — an auth failure, or a config server that never answered (measured: a 60s hang, after which the binary echoes the input config straight back) — threw here. The caller caught the throw and tore the binary down, and the whole run silently continued on the Direct flow instead. The user sees no error; they just quietly lose every binary-flow product.Keeping the default endpoints and warning is strictly better than losing the binary flow.
2. The classic accessibility handler double-scanned every command the binary already wrapped.
getCloudProvider()decides whether a session is a BrowserStack session withhostname.includes('browserstack'). A hub served from a host without that substring is therefore classified as a third-party grid, which makesisBrowserstackSession()false, which makes_isCliAccessibilityFlow()false — soservice.tspicks the classic accessibility handler while the binary is running its CLI accessibility module. Both wrap the same commands.Measured on one such run: 22 classic + 19 CLI scans, where 19 were expected. Duplicate scans inflate
scan_countagainsteffective_expected, which is exactly what the App-A11y stability metric divides — a doubled session reads as an unstable one.Two properties of the fix are deliberate:
browserstack.service.tsmust choose a flow before the binary has necessarily finished booting. On a slow environment that decision lands on the classic handler, the binary then comes up, and a decision cached at session start is already stale by the time a command fires. Re-asking when a scan is about to happen is the only check that can be right.I first wrote this gate using
_isCliAccessibilityFlow()and it never fired, because that predicate inherits the sameisBrowserstackSession()the bug comes from. The registry check is the correction.Not included on purpose: widening
getCloudProvider()itself to recognise internal hosts. That is the root cause, but it changes session classification for every product, not just accessibility — it deserves its own PR and its own blast-radius discussion. This PR makes accessibility correct regardless of how that lands.Related Jira task/s
https://browserstack.atlassian.net/browse/SDK-7422
Release (mandatory for every PR — required for the
ready-for-reviewlabel)Version bump: (required — tick exactly one)
Release notes type: (optional)
Release notes (customer-facing): (optional but encouraged)
Release notes (internal): (required — engineer-facing; what actually changed / why)
cli/index.ts:loadModules()no longer throws on a config with noapisblock — it warns and keeps the default endpoints. Previously the throw tore the binary down and dropped the run to the Direct flow with no user-visible error.service.ts+accessibility-handler.ts: the classic a11y handler now skips a scan when the binary's CLI accessibility module owns the session. The check reads the module registry (hostname-independent) and is evaluated at scan time, because_isCliAccessibilityFlow()inheritsisBrowserstackSession()→getCloudProvider()→hostname.includes('browserstack'), which is false on any non-browserstackhost and caused both flows to wrap the same commands (22 classic + 19 CLI scans measured on one run).Checklist
Verification:
tests/accessibility-handler.test.ts— 48/48 pass, including a new test asserting the command still runs while the duplicate scan is suppressed.tests/service.test.ts+tests/cli/**— 4 failures, identical to untouchedmain(cliUtilsnetwork tests:requestToUpdateCLI×3, stale-binary ×1). This change adds zero failures.Performing scan foris logged before the ownership check, so it overstates the count.PR Validations
Run Tests: Comment RUN_TESTS to trigger sanity tests.