localenv: write [tool.databricks.environment] version on serverless setup-local

[rugpanov]

Summary

When databricks environments setup-local provisions or regenerates pyproject.toml against a serverless target, it now writes a [tool.databricks.environment] section carrying the resolved serverless environment_version:

[tool.databricks.environment]
environment_version = "5"

This lets the same project run interactively, in bundles, and in serverless jobs from one source of truth. It pairs with the VS Code side (DECO-27997), which reads this section as a serverless-version source.

Details

• Serverless only. The version comes from the resolved compute target (--serverless-version, a serverless --job-task, or a serverless bundle target). Cluster targets leave the version empty, so the section is never written and any existing one is left untouched (a no-op).
• Env-owned, formatting-preserving. The new region extends the existing formatting-preserving merge (DECO-27672): environment_version is refreshed in place on regeneration — preserving the line's indentation, any inline comment, and other user keys in the table — inserted when the table exists without it, and the whole table appended when absent. The merge stays idempotent.
• Greenfield. RenderFreshPyproject emits the section for serverless targets.

Testing

• New unit tests in libs/localenv/merge_test.go cover insert/replace/insert-key/cluster-no-op and greenfield rendering, all asserting valid TOML and idempotency.
• Extended the serverless greenfield pipeline test to assert the section end-to-end.
• Regenerated the affected acceptance/localenv goldens; cluster-target acceptance tests are unchanged, confirming the no-op.

DECO-27998

[eng-dev-ecosystem-bot]

Integration test report

Commit: 5a26e8b

Run: 31609389761

	Env	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux		4	4	286	1136	8:34
💚​	aws windows		4	4	288	1134	7:24
🔄​	azure linux	1	4	4	284	1136	10:36
💚​	azure windows		4	4	287	1134	8:05
🔄​	gcp linux	1	1	5	285	1136	9:56
🔄​	gcp windows	1	1	5	287	1134	8:33

10 interesting tests: 4 RECOVERED, 4 SKIP, 2 flaky

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🔄​	TestSyncFullFileSync	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p
🔄​	TestSyncIncrementalFileOverwritesFolder	✅​p	✅​p	✅​p	✅​p	🔄​f	🔄​f
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

Top 10 slowest tests (at least 2 minutes):

duration	env	testname
4:17	gcp linux	TestFilerWorkspaceFilesExtensionsDelete
3:37	gcp windows	TestAccept
2:59	aws windows	TestAccept
2:54	azure windows	TestFilerWorkspaceFilesExtensionsRead
2:54	azure windows	TestAccept
2:41	aws linux	TestFilerRecursiveDelete/workspace_files_extensions
2:39	gcp windows	TestFilerWorkspaceFilesExtensionsRead
2:09	aws windows	TestFilerWorkspaceFilesExtensionsDelete
2:07	aws windows	TestFilerWorkspaceFilesExtensionsReadDir
2:00	azure windows	TestFilerRecursiveDelete/workspace_files

[rclarey]

Why aren't we using a TOML library to do this merging rather than hand-rolling regexes?

Not a blocker since it seems the rest of this file does the same. This is the first of the merging code I'm reviewing so I'm curious why this approach was taken

[anton-107]

Reviewed and probed locally (built the branch, ran the package tests, exercised mergeDatabricksEnvironment against the TOML spellings a real pyproject.toml can use). The core is sound and the change is much smaller in blast radius than #6255 — insert / replace-in-place / insert-key / cluster-no-op all behave as documented, output is valid TOML, and idempotency holds on every shape I tried including the pre-feature upgrade path. Comment and indentation preservation via trailingComment + the captured indent group works.

Three things worth addressing.

1. A non-[table] spelling of the same table produces invalid TOML

mergeDatabricksEnvironment locates the table with tableBounds(lines, "[tool.databricks.environment]"), which only matches a literal bracketed header line. TOML has other ways to define that exact table, and for those the function decides "absent" and appends a second definition. Both of these come out of the merge rejected by the TOML parser:

[tool.databricks]
environment.environment_version = "3"      # dotted key

→ toml: Key 'tool.databricks.environment.environment_version' has already been defined

[tool.databricks]
environment = {environment_version = "3"}  # inline table

→ toml: Key 'tool.databricks.environment' has already been defined

The merged file is then unparseable, so uv sync fails at the provision phase — for a user whose input file was perfectly valid. Note the merge never parses its own output in production (only the tests call requireValidTOML), so nothing catches this before uv does.

To be fair: this hazard is pre-existing for [tool.uv] on main — [tool] + uv = {package = false} already produces the same duplicate-table error today, so the pattern isn't newly introduced by this PR. But this PR adds a second instance of it, and [tool.databricks] with a dotted or inline environment key is a plausible thing for a user (or a future template) to write, more so than an inline [tool.uv]. The cheap fix that fits the existing bail-out philosophy: detect a tool.databricks table containing an environment dotted/inline key and refuse the merge with an errMultilineString-style error telling the user to edit the section manually, rather than emitting a file uv will reject. MergeManaged already has precedent for "refuse rather than risk corrupting" (containsMultilineString, errNoProjectTable).

2. A stale environment_version survives a switch to a cluster target

The no-op-for-cluster behavior is deliberate and documented ("an existing one is left untouched rather than removed"), and there's a test asserting it. But consider the actual sequence: a user runs setup-local --serverless-version 5, gets environment_version = "5" written, then re-targets the project at a cluster (--cluster-id ...). The section stays, still saying 5, now describing a target the project isn't set up for. Since the PR's whole premise is that VS Code and serverless Jobs read this section as a source of truth, a stale value is worse than an absent one — the downstream reader can't tell it's stale.

I don't think silently deleting a user-visible section is right either. But the current behavior deserves at least a warning (the file now carries a serverless version while configured for a cluster), and the trade-off should be stated in the PR description rather than only in a code comment. Right now the description calls it "a no-op", which undersells it.

3. --constraints-only writes the section, unlike every other managed region

acceptance/localenv/constraints-only/output.txt shows the new section landing in constraints-only mode. That's inconsistent with how the mode treats the other env-owned value: dbcPin is explicitly cleared so databricks-connect is "neither written nor asserted", and #6255 extends the same carve-out to its consolidation pass. EnvironmentVersion is passed through unconditionally.

If constraints-only means "manage constraints, don't touch the rest of the project's identity", the environment version arguably belongs behind the same gate. If it's intentional that constraints-only still records the target version, the reasoning should be in the code next to the other mode's comment — currently there's nothing explaining why this region opts out of the carve-out, and the golden change is easy to read as accidental.

Smaller notes

• ServerlessEnvironmentVersion() on *ComputeInfo duplicates the strings.TrimPrefix(c.ServerlessVersion, "v") that Label() already does eight lines above (result.go:181). Worth having Label() call the new method so the "v"-stripping rule lives in one place.
• regionDatabricksEnvironment is inserted between the databricks-connect and tool.uv regions, which reorders the changed region: output in the goldens. Harmless, but it means merge-warnings/output.txt now lists the new region before tool.uv.constraint-dependencies while serverless-check lists it after — the difference comes from which regions changed, not from ordering, so it's fine; just noting it's intentional and not a golden mistake.
• environment_version is written as a quoted string ("5"). Worth confirming with the VS Code side (DECO-27997) that it reads a string and not an integer — the PR body shows the string form, so presumably yes, but a mismatch here is a silent integration break rather than a loud one.
• Tests assert assert.Contains(t, regions, "tool.databricks.environment") with the literal string rather than the regionDatabricksEnvironment constant that's right there in the package. Minor, but the constant is what keeps a rename honest.

Nothing here is a blocker on the mechanics — (1) is the one I'd fix before merge, and (2)/(3) are decisions that should be explicit rather than implicit.

[anton-107]

Approving. The core is sound — insert / replace-in-place / insert-key / cluster-no-op all behave as documented, output is valid TOML, and idempotency holds including on the pre-feature upgrade path.

One thing I'd like fixed before merge, though it's minor and not a regression: a dotted-key or inline-table spelling of [tool.databricks.environment] makes the merge append a second definition and produces invalid TOML that uv sync then rejects. It's the same pre-existing hazard [tool.uv] already has on main, so I'm not blocking on it — but it's cheap to guard following the existing containsMultilineString/errNoProjectTable "refuse rather than corrupt" precedent.

The stale-version-after-cluster-switch behavior and the --constraints-only carve-out are decisions I'd like stated explicitly (in code comments / PR description) rather than left implicit, but they're not blockers. Details in my comment above.