bundle: avoid num_workers with policy defaults

[paddy-devan]

Changes

• Skip implicit num_workers: 0 initialization when cluster policy defaults are enabled.
• Ignore policy-populated autoscale values during direct-engine drift detection for task, for-each-task, and shared job clusters.
• Add table-driven unit coverage and a direct-engine acceptance fixture covering create, plan, and redeploy.

Why

Fixes #3125.

Related precedent: #2755 and databricks/terraform-provider-databricks#4643.

The CLI initializes an omitted job-cluster size to zero to preserve the existing Terraform-provider compatibility behavior. That remains unchanged unless apply_policy_default_values is enabled. In that case, a cluster policy may supply autoscale, and also sending num_workers produces conflicting sizing fields.

This follows the existing precedent that policy-materialized values should not be treated as user-configured drift. The change is limited to the policy-default path; explicit num_workers, explicit autoscale, and configurations with policy defaults disabled retain their current behavior.

Tests

• go test ./bundle/config/mutator/resourcemutator -run TestInitializeNumWorkers -count=1
• go test ./bundle/direct/dresources -count=1
• Manual bundle deployment using the locally built patched CLI
• Added a direct-engine acceptance fixture for create, plan, and idempotent redeploy

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 6230
• Commit SHA: 606abf211dd95ffbcda4ef27fd94e72f6a89ee65

Checks will be approved automatically on success.

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

6 files changed
Suggested: @pietern
Also eligible: @denik, @janniklasrose, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

/bundle/ - needs approval

Files: bundle/config/mutator/resourcemutator/cluster_fixups.go, bundle/config/mutator/resourcemutator/cluster_fixups_test.go, bundle/direct/dresources/resources.yml
Suggested: @pietern
Also eligible: @denik, @janniklasrose, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.}

[paddy-devan]

@andrewnester could you please advise on how I can sign this CLA? 😊 Online I've read that "When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately", but this hasn't happened.

[andrewnester]

@paddy-devan please reach out to dabs-feedback@databricks.com and we'll take it from there