environments setup-local: consolidate databricks-connect to a single managed pin

[rugpanov]

Problem

databricks environments setup-local managed databricks-connect only inside [dependency-groups].dev. When a project (e.g. one generated from the data-engineering template) also pins databricks-connect in [project].dependencies, setup-local set the dev pin to the compute-matched version but left the [project].dependencies pin untouched. uv then had to co-install two disjoint requirements (==15.1.* and >=16.0,<17) and failed with a cryptic unsatisfiable-resolution dump at the provision phase. A warning meant to catch this never fired (it couldn't parse ==X.* wildcards), and warnings are advisory anyway.

Change

In the install flow, after the managed pin lands in [dependency-groups].dev, the merge removes any conflicting databricks-connect pin elsewhere — in [project].dependencies, an optional-dependency extra, or another dependency group. "Conflicting" means the pin's version range provably cannot co-resolve with the managed pin (rangesDisjoint); that is exactly the pin that would make uv sync unsatisfiable. A pin that co-resolves (databricks-connect>=15 vs ~=17.2.0), carries no version, is marker-gated, or already equals the managed version is left in place — so a user declaration that isn't broken is never rewritten, including [project].dependencies, which the default template compiles into the built wheel's metadata. Each removed pin is surfaced with the new W_DBCONNECT_CONSOLIDATED warning.

--constraints-only mode leaves databricks-connect completely untouched. parseClause now models PEP 440 ==X.* prefix-match wildcards, so a conflict between a wildcard user pin and the environment's constraint-dependencies is reported (W_USER_CONSTRAINT_CONFLICT) rather than silently missed.

The removal preserves the file's other bytes: a neighbouring comment is kept, and removing the last element of a multi-line array keeps the closing ] on its own line and preserves the trailing comma. The line-based removal matches the same shapes the rewrite does (double-quoted elements under bare-key arrays); rarer spellings it can't reach — single-quoted pins, quoted TOML keys, inline-table/dotted sub-table forms — are left in place and surfaced as W_DBCONNECT_PIN_DUPLICATED when disjoint.

Testing

• Unit tests (libs/localenv/merge_test.go, warnings_test.go): disjoint pins removed across all three location types; compatible / unversioned / marker-gated / env-equal pins kept; empty-array, trailing-comma, and closing-bracket handling; comment preservation; ==X.* wildcard disjoint math.
• Acceptance: updated merge-warnings / merge-warnings-json / dbconnect-consolidate-check; added constraints-only-existing, wildcard-constraint-conflict-check, and dbconnect-compatible-kept-check (the "kept" behavior).
• Verified end-to-end against the original repro: uv sync resolves cleanly.

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 1deddf8

Run: 31694081825

	Env	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux	4	4	286	1140	5:14
💚​	aws windows	4	4	288	1138	4:00
💚​	azure linux	4	4	285	1140	5:04
💚​	azure windows	4	4	287	1138	3:12
💚​	gcp linux	1	5	286	1140	5:04
💚​	gcp windows	1	5	288	1138	4:17

8 interesting tests: 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:09	gcp windows	TestAccept
3:53	aws windows	TestAccept
3:03	azure windows	TestAccept
2:57	azure linux	TestAccept
2:57	aws linux	TestAccept
2:56	gcp linux	TestAccept

[anton-107]

Reviewed the whole change and probed it locally (built both branches, ran the package tests, and fuzzed removeStrayDatabricksConnect over ~4000 randomized array shapes — single/multi-line, trailing comma or not, comments, extras brackets, include-group tables, single-quoted pins). The mechanics hold up well: every output was valid TOML and idempotent, and I could not break it. The ==X.* wildcard modeling in parseClause/clause.interval is correct, and driving the warnings off planDBConnect rather than a second implementation of the merge's rules is the right call.

My concerns are about the policy the PR adopts, not the parsing.

1. Deleting from [project].dependencies changes what the project ships, not just how it resolves

This is the one I'd want resolved before merge. The stated invariant is "a stray pin in ANY resolvable location makes uv unsatisfiable", but that is only true when the two ranges are disjoint. The code removes unconditionally. Locally, with an env pin of ~=17.2.0:

dependencies = ["databricks-connect>=15"]     # -> dependencies = []
dependencies = ["databricks-connect"]         # -> dependencies = []
dependencies = ["databricks-connect[test]>=15 ; python_version < '3.13'"]  # -> dependencies = []

>=15 co-resolves with ~=17.2.0 at 17.2.x perfectly happily — there is no unsatisfiable resolution to fix, and the pin is still deleted. The marker case is worse: ; python_version < '3.13' means the requirement may not even apply to the resolving interpreter, and splitDepSpec deliberately refuses to compare marker-gated pins for exactly that reason — yet the removal pass deletes it anyway.

And [project].dependencies is not only resolver input. The default template builds a wheel from this same file (build: uv build --wheel in databricks.yml, [build-system] hatchling), so [project].dependencies becomes that wheel's install-requires metadata. A local-dev convenience command silently dropping a runtime dependency from the artifact's published metadata is a different class of edit from retargeting a dev pin — the deployed wheel stops declaring databricks-connect, and nothing in the local uv sync surfaces that.

The [dependency-groups] and [project.optional-dependencies] removals don't have this problem (dev-only, never wheel metadata). I'd suggest either:

• gate the removal on rangesDisjoint(pin, envPin) — only delete what actually breaks resolution, which is also what the PR's own motivating repro (==15.1.* vs >=16,<17) is; or
• keep [project].dependencies out of scope and leave it to W_DBCONNECT_PIN_DUPLICATED, deleting only from the dev-only locations.

Either way this needs an explicit product decision recorded, because "setup-local owns databricks-connect everywhere, including your wheel metadata" is a much larger claim than the bug report requires.

2. Silent deletion when the pin already equals the env pin

if strings.TrimSpace(r.pin) == envPin {
    // Identical to the managed pin: the removal is pure deduplication...
    continue
}

For a second dev-array element this reasoning is right. For [project].dependencies it isn't: dependencies = ["databricks-connect~=17.2.0"] is removed and no warning at all is emitted (I confirmed: only W_REQUIRES_PYTHON_OVERRIDDEN fires). That is not deduplication — it deletes the only declaration in a different table, with the user's file changed and nothing said. If the location isn't [dependency-groups].dev, the pin should be reported even when the strings match.

3. Formatting preservation is weaker than the description claims

The description says "the consolidation preserves the file's other bytes". In practice removing an element reflows the array, because the split-on-commas approach attributes each element's leading whitespace to the previous token:

# in                                   # out
dependencies = [                       dependencies = [
    "databricks-dlt",                      "databricks-dlt",
    "pytest",                              "pytest"]        <- closing bracket pulled up
    "databricks-connect==15.1.*"
]

dependencies = ["databricks-connect==15.1.*", "numpy"]   ->  dependencies = [ "numpy"]   <- stray space

Both are valid TOML and idempotent, so this is cosmetic, not a bug. But the first shape (multi-line, no trailing comma on the last element) is extremely common — it's what uv init and most hand-written pyprojects produce — and the result looks like the tool mangled the file. Worth either fixing (re-emit the element's own leading newline+indent when dropping the last element) or, at minimum, not claiming byte preservation in the description and the removeDbconnectFromArraySpan doc comment.

Relatedly, the carry mechanism relocates comments in a way the comment itself acknowledges ("a comment can be relocated"). I hit it here:

dependencies = [
    "databricks-connect==15.1.*",  # spark session
    "numpy",
]

becomes dependencies = [ # spark session — the comment describing the removed pin migrates onto the opening bracket line. leadingLinesBeforeElement is documented to exclude the element's own trailing comment, and it does; this leaks in through the previous token's split instead. Given the removed element is gone, dropping its own trailing comment would read better than moving it somewhere it no longer describes anything.

4. W_DBCONNECT_PIN_DUPLICATED's doc comment is now narrower than its behavior

// merge can neither rewrite nor remove — a single-quoted element, which the
// double-quoted-only edits do not match — so ...

That names one cause, but the code has at least three (removeStrayDatabricksConnect's own comment lists them: single-quoted, quoted TOML key, inline-table table form), plus the sub-table spellings — I confirmed [dependency-groups.docs] and [project.optional-dependencies.extra] are outside tableBounds' reach, and the userPyprojectTOML doc comment specifically calls the sub-table form out as something users legitimately write. The prose in result.go should list the class ("a spelling the line-based passes don't reach"), not a single example, or it will read as exhaustive and mislead the next reader. Same for the dedented list in dbconnectWarnings.

5. Changelog rationale is stale

No changelog fragment: environments setup-local is not yet released

setup-local shipped in v1.12.0 (2026-08-12) and is in CHANGELOG.md on main — the branch is based on bc0894b16, before that release, which is why the local file still says v1.11.0. Since the command is released, changing what it does to [project].dependencies is user-visible behavior and wants a .nextchanges/cli/ fragment. Please rebase on main and re-check.

Smaller notes

• arrayKeyRe is ^\s*([A-Za-z0-9._-]+)\s*=\s*\[. A dotted key like optional-dependencies.extra = [...] inside [project] matches and is captured as key optional-dependencies.extra, so it isn't equal to "dependencies" and is skipped — correct outcome, but by accident rather than intent. A line of comment would help.
• removeStrayDatabricksConnect returns (lines, nil, false) when len(removed) == 0 after having already built out. Correct, but the discarded out makes it look like a bug; returning early or noting why the built slice is dropped would read better.
• slices.SortFunc(targets, ...) with a.span.start - b.span.start: fine for line indices, but the codebase generally uses cmp.Compare for this. Since the comment already asserts the spans are non-overlapping and already in order per table, a stable sort or cmp.Compare avoids the subtraction idiom.
• The acceptance goldens are good coverage of the happy paths. What's missing is the case in (1): an overlapping stray pin. dbconnect-consolidate-check only uses disjoint pins (==15.1.*, ==14.0.0, ==15.0.0), so the goldens never document the "we delete a pin that resolved fine" behavior — which is exactly the behavior most likely to surprise someone.

The wildcard work and the planDBConnect refactor are both clean and I'd take them as-is. It's the removal scope in (1) plus the silent case in (2) that I think need to change before this lands.

[anton-107]

Requesting changes based on my detailed review above. Two blockers before this can land:

1. Unconditional removal from [project].dependencies. The pin is deleted even when it co-resolves fine with the env pin (e.g. >=15 vs ~=17.2.0), and even when it's marker-gated. Because the default template builds a wheel from this file (uv build --wheel + hatchling), this silently drops a runtime dependency from the shipped artifact's install-requires metadata — nothing in the local uv sync surfaces it. Please gate the removal on rangesDisjoint, or keep [project].dependencies out of scope and leave it to W_DBCONNECT_PIN_DUPLICATED.
2. Silent deletion of an env-identical pin in [project].dependencies (no warning at all). The "pure deduplication" reasoning holds for a second dev-array element, not for the only declaration in another table.

The ==X.* wildcard work and the planDBConnect refactor are clean and I'd keep them as-is. This is a scope decision, not a rewrite. Also please rebase — setup-local shipped in v1.12.0, so the "not yet released, no changelog" rationale is stale and a .nextchanges/cli/ fragment is now warranted.

[rugpanov]

Thanks for the thorough review and the fuzzing — really helpful. All addressed in the latest push (rebased on main):

1 (removal scope) + 2 (env-equal, silent): Gated every removal on rangesDisjoint via a new dbconnectPinConflicts(pin, envPin). A stray databricks-connect pin is now deleted only when its range provably can't co-resolve with the managed pin. So against ~=17.2.0:

• databricks-connect>=15 → kept (co-resolves)
• unversioned databricks-connect → kept
• databricks-connect ; python_version < '3.13' → kept (splitDepSpec refuses markers)
• databricks-connect~=17.2.0 (equal) → kept (not disjoint from itself → no removal, no warning)
• databricks-connect==15.1.* → removed (the motivating repro)

This means [project].dependencies — and the wheel's install-requires metadata the template builds from it — is only ever touched when the project genuinely wouldn't resolve, and the env-equal silent-deletion case is gone. Added dbconnect-compatible-kept-check to lock the "kept" behavior into the goldens.

3 (formatting): Removing the last element of a multi-line array now keeps ] on its own line and preserves the trailing comma (with a guard so an all-removed multi-line array can't produce [,]). The comment-relocation trade-off is documented on removeDbconnectFromArraySpan.

4 (doc): Broadened the W_DBCONNECT_PIN_DUPLICATED / W_DBCONNECT_CONSOLIDATED prose to name the class of spellings the line-based passes don't reach, rather than a single example.

5 (changelog): You're right — setup-local shipped in v1.12.0; the branch was based before that release. Rebased on main and added a .nextchanges/cli/ fragment.

Smaller notes: added the arrayKeyRe dotted-key comment, switched the span sort to cmp.Compare, and noted the discarded out slice. On the overlapping-pin coverage gap — that's the new dbconnect-compatible-kept-check golden.

The ==X.* wildcard modeling and the planDBConnect refactor are unchanged, as you suggested.

[anton-107]

Re-reviewed the three new commits. This addresses everything I raised — verified by building the branch, re-running my probes on the exact cases I flagged, re-fuzzing the consolidation over 6000 randomized array shapes (0 invalid-TOML / 0 non-idempotent), and running the full localenv acceptance suite (61 pass). Clearing my changes-requested.

Blocker 1 — unconditional removal from [project].dependencies: fixed. The removal is now gated on dbconnectPinConflicts → rangesDisjoint. Confirmed each case behaves correctly with env pin ~=17.2.0:

• databricks-connect>=15 → kept (co-resolves at 17.2.x)
• bare databricks-connect (no version) → kept
• databricks-connect>=15 ; python_version < '3.13' (marker-gated) → kept
• databricks-connect==15.1.* (disjoint) → removed, which is the actual bug
• mixed disjoint+overlapping in one array → only the disjoint one goes

The new dbconnect-compatible-kept-check acceptance test covers all three location types (project.dependencies, an optional extra, a sibling group) with compatible/marker/unversioned pins, and asserts no W_DBCONNECT_CONSOLIDATED fires. That's exactly the coverage gap I noted.

Blocker 2 — silent deletion of an env-equal pin: fixed. The special-case continue in dbconnectWarnings is gone, and since an equal pin is never disjoint the gate keeps it entirely — dependencies = ["databricks-connect~=17.2.0"] is now left in place, not silently dropped. Cleaner than warning-on-delete.

Formatting (nit 3): fixed. The trailing-comma / bracket-line restore in removeDbconnectFromArraySpan works — multi-line arrays with the dbc pin last (with or without a trailing comma) now keep ] on its own line and preserve the magic trailing comma. The carry comment-relocation is unchanged and still acceptable.

Doc-comment nits (4): fixed. W_DBCONNECT_PIN_DUPLICATED / W_DBCONNECT_CONSOLIDATED prose now names the class of unreachable spellings; singleClauseRe notes the wildcard; arrayKeyRe documents the dotted-key skip; span sort uses cmp.Compare.

Changelog (5): fixed. Rebased onto main and a .nextchanges/cli/ fragment added, correctly scoped to the conflict-only behavior.

One thing worth a sanity check (not blocking): the consolidation and the duplicate-survivor detection now use two different comparison entry points — dbconnectPinConflicts for what gets deleted, and the existing rangesDisjoint path in the survivor loop for what gets flagged. Both bottom out in rangesDisjoint, so they agree, but they're separate call sites; a future change to one should keep the other in step. The interplay is currently correct — I verified a disjoint single-quoted pin (unreachable by the line-based removal) is still surfaced as W_DBCONNECT_PIN_DUPLICATED, an overlapping unreachable pin stays silent, and a removed-in-docs + duplicated-in-qa case reports both.

Nice work on the turnaround. Approving.