feat(cli): add supabase experiments enable and disable by johnstonmatt · Pull Request #6569 · supabase/cli

johnstonmatt · 2026-09-11T03:49:02Z

Adds supabase experiments enable <feature>… and supabase experiments disable <feature>…, which record an experiment opt-in under [experimental] in the project's own supabase/config.{toml,json}.

Why

Until now the only ways to set one were SUPABASE_EXPERIMENTAL_<NAME>, which lasts a single process, and hand-editing the config file. Hand-editing is easy to get wrong in a way that gives no feedback: appending a second [experimental] header is invalid TOML, and an unparseable config resolves every experiment to off without reporting anything, so the feature simply fails to appear in help or invocation.

The write goes through applyConfigEdits, so it merges into the table the file already has, preserves comments and formatting byte-for-byte, works on config.json as well as config.toml, and refuses a layout it cannot edit safely with a message that names the problem:

cannot write supabase/config.toml: a duplicate table header at experimental —
[experimental] is declared more than once. Merge the duplicate table headers into one, then rerun.

What

command-internal/experiment-registry.ts — the closed set of opt-in booleans (compute, stack) and the source of each one's SUPABASE_EXPERIMENTAL_* name. resolveExperimentalFeature now takes ExperimentName rather than string, so a new gate cannot skip registration, and the registry's one-line descriptions are the FEATURE argument help — the valid names are discoverable from --help instead of only from the source.
commands/experiments/ — the enable/disable pair over a shared engine. Registered unconditionally in cli/root.ts: it is the thing that turns experiments on, so it cannot itself be gated.
Enabling is idempotent and writes nothing when the config already says what was asked; disabling an experiment the config never mentions is likewise a no-op, since an absent key already resolves to off. A SUPABASE_EXPERIMENTAL_* variable that would override what was just written is called out in the output.
-o/--output is rejected in favour of --output-format, following the config diff/config pull precedent for net-new TS-only commands.

Refactor

config pull's edit-refusal phrasing moves to command-internal/config-edit-refusal.ts and takes the command name as a parameter, so both callers share one copy rather than the new command inlining a second.

Reviewer notes

The feature set is deliberately just the two booleans that gate a command path. The other [experimental] keys (orioledb_version, s3_host, pgdelta, …) are configuration a name alone cannot toggle.
disable writes an explicit false rather than removing the key, since applyConfigEdits has no delete operation and an explicit false is the clearer record of intent.

Add `supabase experiments enable <feature>…` and `supabase experiments disable <feature>…`, which record an experiment opt-in under `[experimental]` in the project's own `supabase/config.{toml,json}`. Until now the only ways to set one were an environment variable, which lasts a single process, and hand-editing the config file. Hand-editing is easy to get wrong: appending a second `[experimental]` header is invalid TOML, and an unparseable config resolves every experiment to off with no diagnostic, so the feature just fails to appear. The write goes through `applyConfigEdits`, which merges into the existing table, preserves comments and formatting byte-for-byte, supports `config.json`, and refuses a layout it cannot edit safely with a message naming the problem. Introduce `command-internal/experiment-registry.ts` as the closed set of opt-in booleans (`compute`, `stack`) and the source of each one's `SUPABASE_EXPERIMENTAL_*` name, and narrow `resolveExperimentalFeature` to it so a new gate cannot skip registration. The registry's descriptions are the argument help, so the valid names are discoverable from `--help`. Enabling is idempotent and writes nothing when the config already says what was asked. Disabling an experiment the config never mentions is likewise a no-op, since an absent key already resolves to off. When a `SUPABASE_EXPERIMENTAL_*` variable would override what was just written, the output says so. Hoist `config pull`'s edit-refusal phrasing to `command-internal/config-edit-refusal.ts` and take the command name as a parameter, so both callers share it.

github-actions

🤖 AI Review

All 13 reported findings were verified against the checked-out PR and trusted conventions, producing 10 deduplicated confirmed findings. The most significant bug is that enabling stack in a JSON-backed project reports success but cannot affect backend routing. The remaining findings concern output-flag handling, sanitization, inaccurate help/side-effect documentation, inert telemetry configuration, weak error-message coverage, and comment-policy compliance.

Findings

Severity	Location	Category	Sources	Claim
🟠 MAJOR	`apps/cli/src/commands/experiments/experiments.shared.ts:85`	`correctness`	claude+codex	`experiments enable stack` writes and reports success for `config.json`, but stack backend routing only reads `supabase/config.toml`, so the setting has no effect in JSON-backed projects.
🟡 MINOR	`apps/cli/src/commands/experiments/enable/enable.command.ts:39`	`error-handling`	claude	Both experiment commands omit `outputFormats: GLOBAL_OUTPUT_FORMATS`, so `-o table` and `-o csv` fail in telemetry validation before reaching the intended command-specific refusal.
🟡 MINOR	`apps/cli/src/commands/experiments/experiments.shared.ts:112`	`output-sanitization`	claude	User-controlled refusal paths and environment override values are interpolated into terminal output without the repository's inline-name sanitization.
🟡 MINOR	`apps/cli/src/commands/experiments/experiments.integration.test.ts:223`	`test-coverage`	claude	The new error-message wording is untested; even the test named as verifying that a duplicate header is named only checks failure and file preservation.
🟡 MINOR	`apps/cli/src/commands/experiments/enable/SIDE_EFFECTS.md:19`	`documentation`	codex	The enable and disable side-effect contracts incorrectly promise byte-preserving edits for JSON files.
⚪ NIT	`apps/cli/src/commands/experiments/enable/enable.command.ts:38`	`telemetry`	claude+codex	`safeFlags: ["features"]` is inert: `features` is positional, so selected experiment names are not recorded despite the accompanying comment implying they are logged.
⚪ NIT	`apps/cli/src/commands/experiments/experiments.format.ts:56`	`documentation`	claude	The formatter documentation says notes identify overrides that make the new file value false for the current shell, but a note is emitted for every non-empty override, including one that agrees with the written value.
⚪ NIT	`apps/cli/src/commands/experiments/experiments.command.ts:7`	`documentation`	claude+codex	The experiment family and both subcommand descriptions say settings are written only to `supabase/config.toml`, although the implementation supports and prefers `config.json`.
⚪ NIT	`apps/cli/src/commands/experiments/experiments.integration.test.ts:158`	`code-quality`	codex	The JSON-edit test places its assertion rationale in an inline comment instead of carrying that intent in the test name, contrary to the trusted test-comment convention.
⚪ NIT	`apps/cli/docs/stack-commands.md:16`	`documentation`	claude	The stack documentation still instructs users only to hand-edit `[experimental] stack = true` and omits the new `supabase experiments enable stack` workflow.

Findings outside the diff

⚪ NIT apps/cli/docs/stack-commands.md:16 — The stack documentation still instructs users only to hand-edit [experimental] stack = true and omits the new supabase experiments enable stack workflow.

Stats

Claude findings: 8 · Codex findings: 5 · Confirmed: 10 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

github-actions · 2026-09-11T04:04:09Z

+  Command.withHandler((flags) =>
+    experimentsEnable(flags).pipe(
+      // The feature names are a closed enum, so logging them verbatim carries no user data.
+      withCommandTelemetry({ flags, safeFlags: ["features"] }),


🟡 MINOR · error-handling · source: claude

Both experiment commands omit outputFormats: GLOBAL_OUTPUT_FORMATS, so -o table and -o csv fail in telemetry validation before reaching the intended command-specific refusal.

Evidence: enable.command.ts:39 and disable.command.ts:35 use the default telemetry options; command-telemetry.ts:461-469 defaults to RESOURCE_OUTPUT_FORMATS and validates before the handler, while global-flags.ts:20-28 also permits table and csv.

Suggested fix: Pass outputFormats: GLOBAL_OUTPUT_FORMATS to both telemetry wrappers so every accepted global -o value reaches the handler refusal.

github-actions · 2026-09-11T04:04:09Z

+  const format: ConfigFormat = path.basename(configPath) === "config.json" ? "json" : "toml";
+  const currentText = yield* fs.readFileString(configPath).pipe(
+    Effect.mapError(
+      (cause) =>
+        new ExperimentsConfigReadError({
+          message: `Unable to read ${configPath}: ${cause.message}`,
+        }),
+    ),
+  );
+
+  // `undefined` means the document does not parse; every feature becomes an edit so
+  // `applyConfigEdits` is the one that names what is wrong with it.
+  const previousValues = readExperimentValues(format, currentText);
+  const previousFor = (feature: ExperimentName): boolean => previousValues?.[feature] ?? false;
+  const targets =
+    previousValues === undefined
+      ? features
+      : features.filter((feature) => previousFor(feature) !== input.enabled);
+
+  if (targets.length > 0) {
+    const edits: ReadonlyArray<ConfigEdit> = targets.map((feature) => ({
+      path: ["experimental", feature],
+      value: input.enabled,
+    }));
+    const outcome = applyConfigEdits(currentText, format, edits);


🟠 MAJOR · correctness · source: claude+codex

experiments enable stack writes and reports success for config.json, but stack backend routing only reads supabase/config.toml, so the setting has no effect in JSON-backed projects.

Evidence: experiments.shared.ts:75-109 discovers the authoritative config and edits JSON when selected; packages/config/src/paths.ts:24-38 prefers config.json. stack-backend.ts:108-121 hard-codes supabase/config.toml and never discovers config.json.

Suggested fix: Make stack routing discover and parse both supported config formats, or reject stack edits for JSON-backed projects and test that behavior.

github-actions · 2026-09-11T04:04:09Z

+      const location = refusedPath.length === 0 ? "" : ` at ${refusedPath.join(".")}`;
+      return yield* new ExperimentsUnsupportedLayoutError({
+        message: `cannot write ${configPath}: ${configEditRefusalPhrase(reason)}${location} — ${detail}. ${configEditRefusalRemediation(reason, input.command)}`,


🟡 MINOR · output-sanitization · source: claude

User-controlled refusal paths and environment override values are interpolated into terminal output without the repository's inline-name sanitization.

Evidence: experiments.shared.ts:112 joins the refusal path directly, and experiments.format.ts:75 prints envOverride directly. By contrast, config-pull-run.ts:733 uses configRenderPath, whose implementation at config.format.ts:96-98 applies sanitizeInlineName to prevent control, ANSI, bidi, and forged-line output.

Suggested fix: Sanitize refusal paths, refusal details, and displayed environment values with configRenderPath or sanitizeInlineName.

github-actions · 2026-09-11T04:04:09Z

+  it.live("names the duplicate table header rather than silently leaving the flag off", () =>
+    Effect.gen(function* () {
+      const repo = yield* project(
+        'project_id = "demo"\n\n[experimental]\ncompute = true\n\n[api]\nenabled = true\n\n[experimental]\norioledb_version = ""\n',
+      );
+      const { layer } = setupExperiments({ workdir: repo.dir });
+      const before = yield* repo.read;
+
+      return yield* Effect.gen(function* () {
+        const exit = yield* experimentsEnable({ features: ["stack"] }).pipe(Effect.exit);
+        expect(Exit.isFailure(exit)).toBe(true);
+        expect(yield* repo.read).toBe(before);
+      }).pipe(Effect.provide(layer));
+    }).pipe(Effect.scoped, Effect.provide(BunServices.layer)),


🟡 MINOR · test-coverage · source: claude

The new error-message wording is untested; even the test named as verifying that a duplicate header is named only checks failure and file preservation.

Evidence: experiments.integration.test.ts:223-236 asserts only Exit.isFailure(exit) and unchanged file contents. The project-not-found, unreadable-file, and -o refusal tests at lines 239-279 likewise do not inspect error messages, and no other tests assert the new refusal phrases.

Suggested fix: Assert the typed failure and its message for at least duplicate-header and unsupported-output cases, and add focused coverage for the shared refusal formatter.

github-actions · 2026-09-11T04:04:09Z

+      // The feature names are a closed enum, so logging them verbatim carries no user data.
+      withCommandTelemetry({ flags, safeFlags: ["features"] }),


⚪ NIT · telemetry · source: claude+codex

safeFlags: ["features"] is inert: features is positional, so selected experiment names are not recorded despite the accompanying comment implying they are logged.

Evidence: enable.command.ts:12-16 defines features as an Argument and line 39 lists it in safeFlags; disable repeats this. command-telemetry.ts:148-202 collects only dash-prefixed flags, and lines 212-224 explicitly exclude positional Argument choices.

Suggested fix: Remove the inert safeFlags entry and misleading comment, or add deliberate safe positional telemetry if recording experiment names is required.

github-actions · 2026-09-11T04:04:09Z

+ * The text-mode report: one line per requested experiment, then any environment override that
+ * would make the file's new value a lie for the current shell.


⚪ NIT · documentation · source: claude

The formatter documentation says notes identify overrides that make the new file value false for the current shell, but a note is emitted for every non-empty override, including one that agrees with the written value.

Evidence: experiments.format.ts:56-57 describes a conflicting override, while lines 72-76 emit the note whenever envOverride is defined. experiments.shared.ts:51-54 returns every non-empty override without comparing its boolean meaning to the requested state.

Suggested fix: Describe the note as reporting any active override, or only emit it when the parsed override conflicts with the written value.

github-actions · 2026-09-11T04:04:09Z

+
+export const experimentsCommand = Command.make("experiments").pipe(
+  Command.withDescription(
+    "Manage this project's experimental feature opt-ins, recorded under [experimental] in supabase/config.toml.",


⚪ NIT · documentation · source: claude+codex

The experiment family and both subcommand descriptions say settings are written only to supabase/config.toml, although the implementation supports and prefers config.json.

Evidence: experiments.command.ts:7, enable.command.ts:23, and disable.command.ts:23 name only config.toml; experiments.shared.ts:75-85 uses findCliProjectPaths, and packages/config/src/paths.ts:24-38 prefers config.json when present.

Suggested fix: Mention both supabase/config.toml and supabase/config.json in the family and subcommand help.

github-actions · 2026-09-11T04:04:09Z

+The write is a surgical, format-preserving edit through `applyConfigEdits`: it
+sets `experimental.<feature> = true` inside the existing `[experimental]` table,
+or creates that table when the document has none. Comments, key order, spacing,
+and quoting elsewhere in the file survive byte-for-byte. The file is replaced


🟡 MINOR · documentation · source: codex

The enable and disable side-effect contracts incorrectly promise byte-preserving edits for JSON files.

Evidence: Both SIDE_EFFECTS files state that unrelated comments, key order, spacing, and quoting survive byte-for-byte for config.{toml,json}. packages/config/src/config-edit.ts:1205-1239 parses and fully reserializes changed JSON through JSON.stringify, preserving detected indentation and newline style but normalizing other whitespace and scalar spellings.

Suggested fix: Limit the byte-preservation guarantee to TOML and document JSON reserialization, or implement span-based JSON editing.

github-actions · 2026-09-11T04:04:09Z

+        // Asserted as text, not as a decoded object: the point is that the edit lands inside
+        // the existing object with the file's own indentation intact.


⚪ NIT · code-quality · source: codex

The JSON-edit test places its assertion rationale in an inline comment instead of carrying that intent in the test name, contrary to the trusted test-comment convention.

Evidence: experiments.integration.test.ts:148 names the test only edits a config.json project in place, while lines 158-159 explain why text equality verifies indentation. trusted/CLAUDE.md:276-278 says test names must carry intent and assertion reasoning must not be annotated inline.

Suggested fix: Rename the test to mention preserving the existing JSON object's indentation and remove the assertion comment.

johnstonmatt requested a review from a team as a code owner September 11, 2026 03:49

github-actions Bot reviewed Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(cli): add supabase experiments enable and disable - #6569

feat(cli): add supabase experiments enable and disable#6569
johnstonmatt wants to merge 1 commit into
developfrom
feat/experiments-enable-disable

johnstonmatt commented Sep 11, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

github-actions Bot Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		// The feature names are a closed enum, so logging them verbatim carries no user data.
		withCommandTelemetry({ flags, safeFlags: ["features"] }),

		* The text-mode report: one line per requested experiment, then any environment override that
		* would make the file's new value a lie for the current shell.

		// Asserted as text, not as a decoded object: the point is that the edit lands inside
		// the existing object with the file's own indentation intact.

Uh oh!

Conversation

johnstonmatt commented Sep 11, 2026

Why

What

Refactor

Reviewer notes

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

🤖 AI Review

Findings

Findings outside the diff

Stats

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant