Ignore Pipelines ANSI backend default in direct plans - #6816
Merged
Merged
Conversation
Collaborator
Integration test reportCommit: 068c0c5
Top 3 slowest tests (at least 2 minutes):
|
shreyas-goenka
marked this pull request as ready for review
September 23, 2026 13:41
shreyas-goenka
requested review from
denik,
lennartkats-db and
renaudhartert-db
September 23, 2026 13:41
denik
approved these changes
Sep 23, 2026
denik
left a comment
denik
left a comment
Contributor
There was a problem hiding this comment.
Tests seem unnecessary? We know backend_defaults work and also there are a bunch of tests broken by this so we know that config change is correct.
Contributor
Author
|
@denik I think you are right. Removing all tests added in this PR. |
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Sep 24, 2026
## Release v1.18.0 ### CLI * The AI Runtime commands have moved to `databricks air`. The previous `databricks experimental air` path now directs users to the new command. ([#6722](#6722)) * Write local state, cache, and config files atomically so an interrupted or concurrent write cannot corrupt them. ([#6708](#6708)) * Deprecate `--region` in `databricks auth docker configure` ahead of its removal in the next release, infer the Artifact Registry region when it is omitted, and add `databricks auth docker host --profile <name>` to show the profile's registry host and credential-helper status. ([#6782](#6782)) * Return `UNAUTHENTICATED` instead of `INVALID_REFRESH_TOKEN` when `databricks auth token --output json` cannot refresh a cached U2M token. ([#6731](#6731)) * Retry the current-user (SCIM `Me`) lookup on transient HTTP 500 responses so a temporarily-unavailable backend no longer fails bundle commands outright. ([#6766](#6766)) * Preserve workspace-file and volume access for SSH server descendants when the bootstrap notebook exits and the server survives. ([#6645](#6645)) * `ssh connect` and `ssh setup` now accept a `--keep-detached-processes` flag to keep processes detached from the SSH session (`tmux`, `setsid`, `nohup`) running after the tunnel shuts down. Teardown then terminates only the tunnel's own process group, and the bootstrap job run is held open while any detached process is still running, so the survivors keep their `/Workspace` and `/Volumes` access. A held-open run also suppresses cluster autotermination, so the flag is off by default, is bounded by `--server-timeout`, and is dedicated-cluster only. Without it, the server now logs a warning naming the detached processes it is about to destroy, instead of sweeping them silently. ([#6387](#6387)) ### Bundles * direct: Allow clearing a catalog's or schema's `custom_max_retention_hours` by removing it from configuration. ([#6792](#6792)) * direct: Allow clearing a genie space's `description` and a secret's `comment` by removing them from configuration. ([#6789](#6789)) * Fix direct-engine deploy recreating an MLflow experiment on every deploy when its `trace_location` was set out-of-band. ([#6787](#6787)) * direct: Store a Genie space's `serialized_space` in state as a content hash instead of its full contents. ([#6707](#6707)) * Fix `bundle deploy` failing with "Invalid python file reference" for jobs that use `git_source` with a `spark_python_task` on the direct engine. ([#6751](#6751)) * Fixed the direct engine mishandling UC grants that combine `ALL_PRIVILEGES` with a privilege it does not imply (`MANAGE`, `READ_METADATA`, `EXTERNAL_USE_SCHEMA`, `EXTERNAL_USE_LOCATION`): such privileges were dropped when granted and left behind when revoked, so the deployment never converged. ([#6733](#6733), [#6743](#6743)) * Don't fail migration if clean up actions fail. ([#6772](#6772)) * Ignore the backend-provided `spark.sql.ansi.enabled: "true"` pipeline configuration default when detecting direct-engine drift. ([#6816](#6816)) * Fix recreating a postgres synced table sometimes failing with a 409 ALREADY_EXISTS error while the previous table is still being deleted. ([#6728](#6728)) * Direct engine no longer recreates a resource when an immutable field the config omits was populated by the backend. ([#6790](#6790)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6723](#6723)) * Bump `github.com/databricks/databricks-sdk-go` from v0.178.0 to v0.182.0. ([#6817](#6817)) * Bump the Databricks Terraform provider from v1.132.0 to v1.134.0. ([#6818](#6818))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Treat
spark.sql.ansi.enabled: "true"as a Pipelines backend default in direct-engine drift detection.Why
Pipelines currently adds this default to the user configuration map, causing direct-engine plans to report persistent drift when the bundle omits it.
Tests
task checkspasses.This PR was written by Codex.