Update Databricks CLI to v1.18.0

[eng-dev-ecosystem-bot]

Update Databricks CLI to v1.18.0

[rugpanov]

Executive summary — Databricks CLI v1.17.0 → v1.18.0

Bumps the bundled CLI pin at packages/databricks-vscode/package.json:1888 (cli.version 1.17.0 → 1.18.0). Per CONTRIBUTING.md, remember to re-fetch the binary (yarn workspace databricks run package:cli:fetch) since bin/ is gitignored. Release v1.18.0 delta:

CLI

• AI Runtime commands moved from experimental air to databricks air (#6722).
• Local state, cache, and config files are now written atomically, so an interrupted or concurrent write can't corrupt them (#6708).
• auth docker configure deprecates --region (now inferred), and adds auth docker host to show a profile's registry host / credential-helper status (#6782).
• auth token --output json returns UNAUTHENTICATED instead of INVALID_REFRESH_TOKEN when a cached U2M token can't refresh (#6731).
• Current-user (SCIM Me) lookup is retried on transient HTTP 500s, so a flaky backend no longer fails bundle commands outright (#6766).
• SSH: descendants of the SSH server keep /Workspace + /Volumes access after the bootstrap notebook exits (#6645); new --keep-detached-processes flag for ssh connect / ssh setup keeps tmux/nohup/setsid processes alive past tunnel teardown (off by default, dedicated-cluster only, bounded by --server-timeout) (#6387).

Bundles — all fixes are on the direct deployment engine:

• Allow clearing fields by removing them from config: custom_max_retention_hours (#6792), genie description / secret comment (#6789).
• Stop spurious recreation: MLflow experiment with out-of-band trace_location (#6787), resources whose backend-populated immutable field is omitted from config (#6790); ignore backend spark.sql.ansi.enabled default when detecting drift (#6816).
• Fix bundle deploy "Invalid python file reference" for jobs using git_source + spark_python_task (#6751).
• Fix ALL_PRIVILEGES + non-implied privilege (MANAGE, READ_METADATA, EXTERNAL_USE_*) never converging (#6733, #6743).
• Store genie serialized_space as a content hash in state (#6707); don't fail migration on cleanup errors (#6772); fix postgres synced-table recreate racing a 409 ALREADY_EXISTS (#6728).

Dependency updates

• Vulnerability bumps (#6723); databricks-sdk-go v0.178.0 → v0.182.0 (#6817); Terraform provider v1.132.0 → v1.134.0 (#6818).

What could be integrated into the VS Code extension experience

Honest headline: this is a maintenance bump with essentially no net-new integration surface. The extension shells the CLI for a narrow set of subcommands — auth profiles (CliWrapper.ts:693), bundle validate|summary|deploy|destroy|sync|init|schema, ssh connect (CliWrapper.ts:147), and aitools — and authenticates through the TypeScript SDK, not CLI auth commands. Ranked by effort-to-value:

1. Reinforce the direct-engine migration nudge — transparent, zero code (highest value).
The extension already steers users off the Terraform engine: BundleEngineManager raises a "migrate to the direct deployment engine" warning with a migration-guide link whenever validate output reports engine === "terraform" (BundleEngineManager.ts:18-21,74; engine read at BundleValidateModel.ts:17-19,93-94; docs URL BundleEngineManager.ts:11). Every Bundles fix in this release hardens the exact engine the extension recommends, and they land transparently through the commands the extension already runs — bundle validate (CliWrapper.ts:1003), bundle summary --force-pull (CliWrapper.ts:1017), bundle deploy (CliWrapper.ts:1153). Two are directly visible in the extension's UI: the git_source + spark_python_task deploy fix (#6751) affects task types the resource explorer renders (TaskTreeNode.ts:123), and the non-converging-grant fix (#6733) removes a class of "deploy never finishes" reports. No action needed beyond confirming the warning copy/URL still reads correctly — but this is the item worth being aware of, because it makes the path the extension pushes people toward materially more reliable.

2. Atomic local-state writes — transparent robustness win, zero code.
The extension force-kills the entire CLI process tree on cancellation (taskkill /T /F on Windows, group SIGTERM elsewhere — cliProcess.ts:67-82), and cancellation tokens are wired through bundleDeploy/bundleDestroy/bundleSync (CliWrapper.ts:1160,1197,1231). Before #6708, killing a deploy/sync mid-write could leave .databricks/ state torn; now those writes are atomic. No action needed — pure reliability improvement for a thing the extension does routinely.

3. SCIM Me retry on 500 — transparent, zero code.
#6766 retries the current-user lookup inside bundle commands, which is what the extension drives. Note the extension's own current-user resolution goes through the SDK (DatabricksWorkspace.ts:93, AuthProvider.ts:281,491), so that path is unaffected — but the CLI-internal lookup during bundle validate/deploy/summary gets more resilient. No action needed.

4. --keep-detached-processes for ssh connect — net-new feature work, speculative value (lowest).
The extension builds its ssh connect line in getSshConnectCommand (CliWrapper.ts:143-157) and passes only --ide, --auto-approve, and compute flags — there's no concept of detached/persistent remote processes in the SSH UI (SshCommands.ts). Wiring #6387 would be genuinely net-new (a new arg plus a UI affordance and the autotermination-suppression caveat to surface), and it's unclear the extension's "open a remote window" flow wants it. Flagging only for completeness; I wouldn't pursue it without a concrete user need.

No-ops for the extension (explicitly not applicable):

• experimental air → air (#6722) — the extension never invokes air; it uses aitools. No-op.
• auth docker deprecation / auth docker host (#6782) — no auth docker call anywhere in src. No-op.
• auth token --output json error code (#6731) — the extension never parses auth token JSON; auth is SDK-driven and the only CLI auth call is auth profiles (CliWrapper.ts:693). No-op.
• SSH descendant workspace-file access (#6645) — server-side lifecycle behind ssh connect; transparent, no client change.
• Dependency bumps (#6723, #6817, #6818) — the databricks-sdk-go and Terraform-provider bumps are internal to the CLI binary; the extension ships its own TypeScript SDK and its own terraformMetadata (extension.ts:524-537), so these ride along with the re-fetched binary and require no change here.

_{🤖 Auto-generated executive summary of the CLI v1.17.0 → v1.18.0 changelog. Integration notes are opportunities, not commitments — verify before acting.}

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2209
• Commit SHA: fdacc9d4ab0e54b3e3cd533dfd1b9d00669967b5

Checks will be approved automatically on success.