Skip to content

fix(a11y): keep the binary flow alive on a degenerate config and stop double scanning - #186

Open
kamal-kaur04 wants to merge 2 commits into
mainfrom
SDK-7422-cli-flow-fixes
Open

fix(a11y): keep the binary flow alive on a degenerate config and stop double scanning#186
kamal-kaur04 wants to merge 2 commits into
mainfrom
SDK-7422-cli-flow-fixes

Conversation

@kamal-kaur04

Copy link
Copy Markdown
Collaborator

What is this about?

Two defects that only surface together, and only away from production hostnames. Both were found while verifying SDK-7422 on an internal environment; neither is specific to that environment, and the second one corrupts a production metric.

1. A config with no apis block killed the binary flow entirely.

loadModules() dereferenced this.config.apis unconditionally. A degenerate config — an auth failure, or a config server that never answered (measured: a 60s hang, after which the binary echoes the input config straight back) — threw here. The caller caught the throw and tore the binary down, and the whole run silently continued on the Direct flow instead. The user sees no error; they just quietly lose every binary-flow product.

Keeping the default endpoints and warning is strictly better than losing the binary flow.

2. The classic accessibility handler double-scanned every command the binary already wrapped.

getCloudProvider() decides whether a session is a BrowserStack session with hostname.includes('browserstack'). A hub served from a host without that substring is therefore classified as a third-party grid, which makes isBrowserstackSession() false, which makes _isCliAccessibilityFlow() false — so service.ts picks the classic accessibility handler while the binary is running its CLI accessibility module. Both wrap the same commands.

Measured on one such run: 22 classic + 19 CLI scans, where 19 were expected. Duplicate scans inflate scan_count against effective_expected, which is exactly what the App-A11y stability metric divides — a doubled session reads as an unstable one.

Two properties of the fix are deliberate:

  • It asks the CLI module registry, not the hostname. The accessibility module exists only when the binary owns accessibility for this session, which is the actual question. Hostname is a proxy that is wrong on any host not literally containing browserstack.
  • It is evaluated at SCAN time, not at session start. service.ts must choose a flow before the binary has necessarily finished booting. On a slow environment that decision lands on the classic handler, the binary then comes up, and a decision cached at session start is already stale by the time a command fires. Re-asking when a scan is about to happen is the only check that can be right.

I first wrote this gate using _isCliAccessibilityFlow() and it never fired, because that predicate inherits the same isBrowserstackSession() the bug comes from. The registry check is the correction.

Not included on purpose: widening getCloudProvider() itself to recognise internal hosts. That is the root cause, but it changes session classification for every product, not just accessibility — it deserves its own PR and its own blast-radius discussion. This PR makes accessibility correct regardless of how that lands.

Related Jira task/s

https://browserstack.atlassian.net/browse/SDK-7422

Release (mandatory for every PR — required for the ready-for-review label)

Version bump: (required — tick exactly one)

  • minor (backwards-compatible feature)
  • patch (bug fix or other small change)

Release notes type: (optional)

  • New Feature
  • Bug Fix
  • Other Improvement

Release notes (customer-facing): (optional but encouraged)

  • Fixed accessibility scans being sent twice for a single command in some environments, which made scan counts inaccurate.
  • Fixed the SDK silently falling back to a non-binary flow when the configuration response was incomplete.

Release notes (internal): (required — engineer-facing; what actually changed / why)

  • cli/index.ts: loadModules() no longer throws on a config with no apis block — it warns and keeps the default endpoints. Previously the throw tore the binary down and dropped the run to the Direct flow with no user-visible error.
  • service.ts + accessibility-handler.ts: the classic a11y handler now skips a scan when the binary's CLI accessibility module owns the session. The check reads the module registry (hostname-independent) and is evaluated at scan time, because _isCliAccessibilityFlow() inherits isBrowserstackSession()getCloudProvider()hostname.includes('browserstack'), which is false on any non-browserstack host and caused both flows to wrap the same commands (22 classic + 19 CLI scans measured on one run).

Checklist

  • Ready to review
  • Has it been tested locally?

Verification:

  • tests/accessibility-handler.test.ts48/48 pass, including a new test asserting the command still runs while the duplicate scan is suppressed.
  • tests/service.test.ts + tests/cli/** — 4 failures, identical to untouched main (cliUtils network tests: requestToUpdateCLI ×3, stale-binary ×1). This change adds zero failures.
  • The double-scan behaviour was confirmed on the wire (session-log scan census), not from SDK log lines — Performing scan for is logged before the ownership check, so it overstates the count.

PR Validations

Run Tests: Comment RUN_TESTS to trigger sanity tests.

… double scanning

Two defects that only surface together, and only away from production hostnames.

1. loadModules() dereferenced config.apis unconditionally. A config with no apis
   block — an auth failure, or a config server that never answered (measured: a
   60s hang, after which the binary echoes the input config back) — threw here,
   the caller tore the binary down, and the entire run silently fell through to
   the Direct flow. Keeping the default endpoints is strictly better than losing
   the binary flow.

2. getCloudProvider() classifies a session by hostname.includes('browserstack'),
   so a hub served from a host without that substring reads as a third-party
   grid. isBrowserstackSession() then goes false, _isCliAccessibilityFlow() with
   it, and the service runs the classic accessibility handler WHILE the binary
   runs its CLI accessibility module — both wrapping the same commands. Measured
   on one such run: 22 classic + 19 CLI scans where 19 were expected. Duplicate
   scans inflate scan_count against effective_expected and corrupt the App-A11y
   stability metric.

   The gate asks the CLI module registry rather than the hostname, and asks at
   SCAN time rather than session start: service.ts must choose a flow before the
   binary has necessarily finished booting, so a decision made then can be stale
   by the time a command fires.

Verified: 48/48 accessibility-handler tests pass. service + cli suites show 4
failures, identical to untouched main (cliUtils network tests).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kamal-kaur04
kamal-kaur04 requested a review from a team as a code owner September 8, 2026 17:11
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited)

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 937fe763-f5e4-40d9-978a-4af602229d0b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant