Skip to content

Add ucode setup: interactive managed-config authoring - #268

Open
tt-le wants to merge 3 commits into
tien/managed-setup-serializerfrom
tien/managed-setup-wizard
Open

Add ucode setup: interactive managed-config authoring#268
tt-le wants to merge 3 commits into
tien/managed-setup-serializerfrom
tien/managed-setup-wizard

Conversation

@tt-le

@tt-le tt-le commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Demo

Screen.Recording.2026-08-05.at.10.11.05.AM.mov

Changes

Stacked on #267 — review that one first. This PR's diff includes it until #267 merges.

The admin-facing wizard on top of #267's serializer. Walks a workspace admin through agents, per-agent models, tracing, MCP servers, skills, and a spend-routing budget policy, then writes the manifest to ~/.ucode/managed-settings.json. Publishing is a separate ucode apply (next PR) so an admin can review the file first — and ucode setup show prints both the summary and the exact proto-JSON that apply will POST.

managed_wizard.py is the interaction layer only; catalogs, validation, and serialization stay in managed_setup.py. Sub-flows an admin already knows are delegated to the existing commands (configure_tracing_command, configure_mcp_command, configure_skills_mcp_command) and their results read back out of state.json, so there is still exactly one picker per concern.

Claude Code is prompted per model family

ClaudeModelConfig.models is a ClaudeDefaultModels message — four optional slots (default_opus_model / default_sonnet_model / default_haiku_model / default_fable_model), since Claude Code addresses models by family alias rather than from a list. A flat multi-select cannot express that.

? Default opus model:
  › system.ai.claude-opus-5
    system.ai.claude-opus-4-8
    system.ai.claude-opus-4-7
    ... (skip opus)

Each family is skippable (the slots are optional, and an unset one falls back to default_model); the overall default_model is then chosen from the slots that were filled, so it can never name a model the config doesn't carry. With one slot filled there is nothing to choose, so it reports what it inferred rather than asking a single-option question.

That required looking past state["claude_models"], which is {family: newest} because the launch path pins one model per alias — the workspace tested against has 12 Claude models and the bucketed view shows 4. discover_claude_models_unbucketed returns the full listing, claude_family_candidates groups it by family, and _known_models reads the same set so validation cannot reject a version the prompts offered. discover_model_services is untouched — every agent's config writer depends on the shape of claude_models.

Other model-prompt behaviour

Provider-service agents get a real picker. list_model_provider_services already returns each service's config.targets; the wizard now selects the whole service dict and offers those, instead of an empty text prompt that required knowing a provider-side id by heart. allow_all_targets and relayed Anthropic subscription services have nothing to enumerate, so those fall back to free text — but say why and give an example.

Gemini and Copilot get one model, not a multi-select: their writers take a single model and set one env var, so extra picks were silently discarded at apply time. Wizard-level only — both protos declare repeated string models, so --from-file still serializes a list. OpenCode and Pi keep their multi-select; they really do show a model picker.

Nothing is pre-checked. options[0] is whatever discovery sorted first, not a recommendation — for Pi a Claude model, for Codex the oldest GPT. It only existed to make Enter work, which stopped mattering once every agent required a model: an empty selection is re-prompted. That also makes _prompt_models_for_agent total. Without it, picking a multi-model agent and hitting Enter produced an agent with no model_config, and if it was the default_agent the run died at the end — after tracing, MCP, skills, and budget had all been answered.

Budget tiers offer only the agent's own models

The picker previously read the workspace catalog and called load_state(), so it could not have seen this run's choices even in principle — configuring Pi with one Kimi model still offered every family. Picking one of the extras was not harmless: validateStoredConfig checks only that a tier's default_agent is in enabled_agents, never that the agent has the model, so the tier would activate and hand a developer a model their agent was never given. validate_manifest gains that missing rule, which is the durable half — it guards --from-file and hand-written manifests too.

Tiers are prompted in percent and stored as fractions, keeping the spec-vs-API units mismatch in one place.

Smaller notes

Model ids stay bare (system.ai.claude-opus-4-8), not provider-prefixed: each agent's writer adds whatever its config format needs, so the manifest stays agent-neutral.

Two readback details: tracing is stored as tracing.uc_destination but the managed config calls it tracing.table, and an enabled-but-not-UC-backed experiment has no table so it is omitted rather than published empty; state.json records each MCP server's URL but not its type, so _mcp_type_for_url maps the URL shape back and skips anything unrecognized instead of guessing.

databricks.py adds is_workspace_admin (SCIM Me group membership) and list_workspace_budgets (workspace-scoped — ucode never creates budgets, an admin picks an existing one). The admin gate errors on a definite non-admin and warns-then-continues when the check itself fails, since the API enforces the same rule server-side.

Testing

uv run pytest — 1334 passed, 36 skipped. 116 new cases. (tests/test_e2e_user_agent.py::test_user_agent_arrives_at_gateway fails on my machine because /etc/claude-code/managed-settings.json overrides ANTHROPIC_BASE_URL past the test's capture server; it fails on main too and is unrelated.)

Several are mutation-checked: reverting the claude flat multi-select fails eight, reverting the budget-tier picker fails test_offers_only_the_models_the_agent_was_configured_with, and dropping the tier-model validation rule fails test_tier_model_must_be_one_the_agent_has.

Verified interactively against a staging workspace: six opus versions and four sonnet offered, a skipped family absent from models, and the result round-trips through normalize_managed_config to the proto shape.

Known gap

discover_model_services and discover_claude_models_unbucketed both page /api/2.1/unity-catalog/model-services, so a run that configures Claude lists it twice (one extra spinner). _claude_candidates caches on state["all_claude_models"], bounding it to once per run. Removing the second listing means widening discover_model_services' 5-tuple return, which is asserted in eight tests and mocked in sixteen — left for a follow-up rather than growing this PR.

This pull request and its description were written by Isaac.

tt-le added 2 commits August 5, 2026 17:52
The admin-facing wizard on top of the serializer added in the parent commit.
Walks a workspace admin through agents, per-agent models, tracing, MCP servers,
skills, and a spend-routing budget policy, then writes the manifest to
`~/.ucode/managed-settings.json`. Publishing is a separate `ucode apply` (next
change) so an admin can review the file first — and `ucode setup show` prints both
the summary and the exact proto-JSON that apply will POST.

`managed_wizard.py` is the interaction layer only; catalogs, validation, and
serialization stay in `managed_setup.py`. Sub-flows an admin already knows are
delegated to the existing commands (`configure_tracing_command`,
`configure_mcp_command`, `configure_skills_mcp_command`) and their results read back
out of `state.json`, so there is still exactly one picker per concern.

Claude Code is prompted per model family, not as a flat list. `ClaudeModelConfig.models`
is a `ClaudeDefaultModels` message — four optional slots (`default_opus_model` /
`default_sonnet_model` / `default_haiku_model` / `default_fable_model`), since Claude
Code addresses models by family alias rather than from a list, and a flat multi-select
cannot express that. Each family is skippable (the slots are `optional`, and an unset
one falls back to `default_model`); the overall `default_model` is then chosen from the
slots that were filled, so it can never name a model the config doesn't carry. With one
slot filled there is nothing to choose, so it reports what it inferred rather than
asking a single-option question.

That required looking past `state["claude_models"]`, which is `{family: newest}` because
the launch path pins one model per alias — the workspace tested against has 12 Claude
models and the bucketed view shows 4. `discover_claude_models_unbucketed` returns the
full `system.ai.claude-*` listing, `claude_family_candidates` groups it by family, and
`_known_models` reads the same set so validation cannot reject a version the prompts
offered. `discover_model_services` is untouched — every agent's config writer depends on
the shape of `claude_models`.

Provider-service agents get a real picker too. `list_model_provider_services` already
returns each service's `config.targets` (the provider-side model ids); the wizard now
selects the whole service dict and offers those, instead of an empty text prompt that
required knowing a provider-side id by heart. Two cases legitimately have nothing to
enumerate — `allow_all_targets`, and a relayed Anthropic subscription service that
routes by canonical name — and those fall back to free text, but say why and give an
example.

Gemini and Copilot get one model rather than a multi-select: their writers take a single
model and set one env var, so extra picks were silently discarded at apply time. This is
a wizard restriction, not a serializer one — both protos do declare `repeated string
models`, so `_FLAT_MODEL_LIST_AGENTS` is unchanged and a hand-written `--from-file`
manifest still serializes a list. OpenCode and Pi keep their multi-select; they really
do show a model picker.

Nothing is pre-checked in the model multi-selects. `options[0]` is whatever discovery
sorted first, not a recommendation — for Pi a Claude model, for Codex the oldest GPT —
so pre-checking it made "hit Enter" produce an arbitrary config and read as an
endorsement. It only existed to make Enter work at all, which stopped mattering once
every agent required a model: an empty selection is re-prompted
(`_require_selection` / `_require_multi_selection` / `_require_text`), which also makes
`_prompt_models_for_agent` total. Without that, picking a multi-model agent and hitting
Enter produced an agent with no `model_config`, and if it was the `default_agent` the run
died at the *end* — after tracing, MCP, skills, and budget had all been answered. Ctrl-C
still aborts.

Budget tiers offer only the models their agent was configured with, via
`configured_models_for_agent`. The picker previously read the workspace catalog and
called `load_state()`, so it could not have seen this run's choices even in principle —
configuring Pi with one Kimi model still offered every family. Picking one of the extras
was not harmless: `validateStoredConfig` checks only that a tier's `default_agent` is in
`enabled_agents`, never that the agent has the model, so the tier would activate and hand
a developer a model their agent was never given. `validate_manifest` gains that missing
rule, which is the durable half — it guards `--from-file` and hand-written manifests too.
Tiers are prompted in percent and stored as fractions (`prompt_for_percentage` converts),
keeping the spec-vs-API units mismatch in one place.

Model ids stay bare (`system.ai.claude-opus-4-8`), not provider-prefixed: each agent's
writer adds whatever its config format needs (see `opencode._resolve_model_selector`), so
the manifest stays agent-neutral and matches the spec's examples.

Two readback details worth noting:
- Tracing is stored as `tracing.uc_destination` but the managed config calls it
  `tracing.table`; an enabled-but-not-UC-backed experiment has no table, so it is
  omitted rather than published empty.
- `state.json` records each MCP server's resolved URL but not its type, while the
  manifest needs `{name, type}`. `_mcp_type_for_url` maps the URL shape back to a type
  and skips anything unrecognized instead of guessing.

`databricks.py`: `is_workspace_admin` (SCIM `Me` group membership) and
`list_workspace_budgets` (workspace-scoped, so no account auth is needed — ucode never
creates budgets, an admin picks an existing one). The admin gate errors on a definite
non-admin and warns-then-continues when the check itself fails, since the API enforces
the same rule server-side.

`ui.py`: `preselected`/`prompt` on `prompt_for_tools`, plus `prompt_for_multi_selection`,
`prompt_for_text`, and `prompt_for_percentage`. Reuses the existing
`prompt_yes_no_default` rather than adding a second defaulting yes/no.

README gains a "Managed config for a workspace (admins)" section, `setup` rows in Other
Commands, and `~/.ucode/managed-settings.json` in Managed Local Files. It states plainly
that the wizard leaves your agent configs alone *except* when you accept the tracing /
MCP / skills sub-steps, which do configure this machine — the one thing about the flow
that isn't obvious.

Verified interactively against eng-ml-inference.staging: six opus versions and four
sonnet offered, a skipped family absent from `models`, and the result round-trips through
`normalize_managed_config` to the proto shape.

Tests: 116 cases. Several are mutation-checked — reverting the claude flat multi-select
fails eight, reverting the budget-tier picker fails
`test_offers_only_the_models_the_agent_was_configured_with`, and dropping the tier-model
validation rule fails `test_tier_model_must_be_one_the_agent_has`. Note `typer.Exit`
subclasses RuntimeError, so raising it inside the command's try block made a successful
run print "ERROR 0"; fixed by exiting after the handler, with
`test_successful_setup_exits_zero` failing if that regresses.

Known gap: `discover_model_services` and `discover_claude_models_unbucketed` both page
`/api/2.1/unity-catalog/model-services`, so a run that configures Claude lists it twice.
`_claude_candidates` caches on `state["all_claude_models"]`, which bounds it to once per
run, but removing the second listing means widening `discover_model_services`' 5-tuple
return — asserted in eight tests and mocked in sixteen — so it is left for a follow-up.

Co-authored-by: Isaac
Two CI failures, both passing locally for environment reasons.

`_claude_candidates` guarded its catalog fetch with `except RuntimeError`, but
`get_databricks_token` shells out to `databricks auth token` — so a machine
without the CLI on PATH raises `FileNotFoundError`, which is an `OSError` and
sailed straight through. This is a real bug, not just a test artifact: any user
without the Databricks CLI installed hit an uncaught traceback mid-wizard
instead of degrading to the bucketed per-family picks. Now catches `OSError`
too. `test_claude_candidates_survive_a_missing_databricks_cli` covers it, and
`test_every_agent_always_gets_a_default_model` — which reached the network on
its claude pass — now stubs the fetch rather than depending on the host.

`test_setup_help_lists_from_file` grepped `--from-file` out of rendered `--help`
output. Rich ellipsizes option names to fit the terminal, so below roughly 40
columns it renders `--fro…`; hosted runners report no width and get the narrow
fallback. It now asserts on the declared Click option, which no amount of
wrapping can change, and still checks the help renders successfully.

Verified by reverting each fix in turn: the first fails with the same
`FileNotFoundError: 'databricks'` CI reported, the second with the same missing
`--from-file` assertion at `COLUMNS=30`.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from 48d9066 to b078138 Compare August 5, 2026 17:53

@AarushiShah-db AarushiShah-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some high level comments based on the video

  • Configuration Summary - can we put a box around it
Image
  • Dry run: is there a reason why its ucode configure --dry-run? Can it just be ucode —dry-run?
  • Warning message update: "A published Claude configuration already exists for this workspace. Publishing will replace it. Make sure this configuration includes everything you want to keep."
  • For the model picker , can we add option to filter by search (you type and it will filter the models) Same with budget policy tracker
  • What is the initial "Fetching available models" step doing and is that needed? because aren't we doing individual API calls anyways to get the models for the individual agents?

tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together,
down from 24.

Tests: +9. Mutation-verified — dropping the cache read fails two cache tests,
dropping `searchable` fails the picker test, and un-boxing the summary fails the
panel test.

Deferred, with Aarushi's agreement: the dry-run hint still points at
`ucode configure --dry-run`. The intent is a root `ucode --dry-run` that launches
the default agent against the authored managed config, which needs both that flag
and bare-`ucode` launch behavior that doesn't exist yet.

Co-authored-by: Isaac
tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together,
down from 24.

Tests: +9. Mutation-verified — dropping the cache read fails two cache tests,
dropping `searchable` fails the picker test, and un-boxing the summary fails the
panel test.

Deferred, with Aarushi's agreement: the dry-run hint still points at
`ucode configure --dry-run`. The intent is a root `ucode --dry-run` that launches
the default agent against the authored managed config, which needs both that flag
and bare-`ucode` launch behavior that doesn't exist yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from 7fff00b to b8e191a Compare August 5, 2026 20:49
tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together,
down from 24.

Tests: +9. Mutation-verified — dropping the cache read fails two cache tests,
dropping `searchable` fails the picker test, and un-boxing the summary fails the
panel test.

Two more redundant lines removed while in here. The "Next steps" block no longer
suggests `ucode configure --dry-run`: the manifest describes what *developers*
should get, while that command previews *this machine's* agent configs, so it
implied a local test it does not perform. Publishing is the only next step there is
today. And the wizard's own `ensure_databricks_auth` is now quiet, because
`configure_shared_state` authenticates a moment later and prints the same "auth
already available" success — the wizard's call still has to run first, since the
admin gate and the existing-config check both need a token before discovery.

Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the
default agent against the authored config, so an admin can try it locally before
publishing. That needs both the flag and bare-`ucode` launch behavior, neither of
which exists yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from b8e191a to 57cb08d Compare August 5, 2026 21:03
tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together,
down from 24.

Tests: +9. Mutation-verified — dropping the cache read fails two cache tests,
dropping `searchable` fails the picker test, and un-boxing the summary fails the
panel test.

Two more redundant lines removed while in here. The "Next steps" block no longer
suggests `ucode configure --dry-run`: the manifest describes what *developers*
should get, while that command previews *this machine's* agent configs, so it
implied a local test it does not perform. Publishing is the only next step there is
today. And the wizard's own `ensure_databricks_auth` is now quiet, because
`configure_shared_state` authenticates a moment later and prints the same "auth
already available" success — the wizard's call still has to run first, since the
admin gate and the existing-config check both need a token before discovery.

Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the
default agent against the authored config, so an admin can try it locally before
publishing. That needs both the flag and bare-`ucode` launch behavior, neither of
which exists yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from 57cb08d to c536a37 Compare August 5, 2026 21:12
tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together,
down from 24.

Tests: +9. Mutation-verified — dropping the cache read fails two cache tests,
dropping `searchable` fails the picker test, and un-boxing the summary fails the
panel test.

Two more redundant lines removed while in here. The "Next steps" block no longer
suggests `ucode configure --dry-run`: the manifest describes what *developers*
should get, while that command previews *this machine's* agent configs, so it
implied a local test it does not perform. Publishing is the only next step there is
today. And the wizard's own `ensure_databricks_auth` is now quiet, because
`configure_shared_state` authenticates a moment later and prints the same "auth
already available" success — the wizard's call still has to run first, since the
admin gate and the existing-config check both need a token before discovery.

Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the
default agent against the authored config, so an admin can try it locally before
publishing. That needs both the flag and bare-`ucode` launch behavior, neither of
which exists yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from c536a37 to 0d02f98 Compare August 5, 2026 21:21
tt-le added a commit that referenced this pull request Aug 5, 2026
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The existing-config warning also says what it is warning about. It named the
enabled agents in a parenthetical — "already exists for this workspace (claude)" —
which read as though the workspace itself were called claude, and listing the
contents implied the config was per-agent when there is exactly one per workspace
covering agents, MCP servers, skills, tracing, and the budget policy. It now says
that scope plainly and itemizes nothing: an inventory doesn't change what the admin
should do, and `ucode setup show` prints the real thing for anyone comparing.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

The Model Provider Service listing gets the same treatment, for the same reason: it
is workspace-wide and filtered per agent afterwards, so the wizard re-listed it once
per MPS-capable agent. One call now serves them all. Cached entries are copied on
both store and read, since the wizard treats the list as its own.

Measured against eng-ml-inference.staging: 12 HTTP pages for the model catalog
across both callers, down from 24; and one MPS listing for claude + codex, down from
two.

Tests: +15. Mutation-verified — dropping either cache read fails a cache test,
dropping `searchable` fails the picker test, un-boxing the summary fails the panel
test, and re-adding the config inventory fails the warning test. The MPS aliasing
guard is split in two on purpose: the store-side and read-side copies each need
their own case, since a single test passed with either one still present.

Two more redundant lines removed while in here. The "Next steps" block no longer
suggests `ucode configure --dry-run`: the manifest describes what *developers*
should get, while that command previews *this machine's* agent configs, so it
implied a local test it does not perform. Publishing is the only next step there is
today. And the wizard's own `ensure_databricks_auth` is now quiet, because
`configure_shared_state` authenticates a moment later and prints the same "auth
already available" success — the wizard's call still has to run first, since the
admin gate and the existing-config check both need a token before discovery.

Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the
default agent against the authored config, so an admin can try it locally before
publishing. That needs both the flag and bare-`ucode` launch behavior, neither of
which exists yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from 0d02f98 to 7f8ac63 Compare August 5, 2026 22:13
Review feedback on #268.

The configuration summary is boxed. It is the one block an admin is meant to read
as a whole and check against what they intended, and it arrives after a long flow
of prompts, where loose key/value lines blend into everything printed before them.
`ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare
title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one.

Every picker in the flow now filters as you type. On the workspace this was built
against, the model lists run to 16 GPT entries and 12 Claude ones, which is more
than is comfortable to arrow through. questionary supports this directly, with one
constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys`
(j and k are search characters), so search costs j/k navigation. Arrow keys still
work, and it is opt-in via a `searchable` flag so short pickers — "Databricks
Hosted vs External Models" — keep j/k.

The existing-config warning now says what actually happens. It claimed "there is
no partial update yet", which was both vague and, once `ucode apply` PATCHes, no
longer true. Publishing is a full replace of the workspace's config, so the warning
says that plainly and names the agents already published, which is what the admin
stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to
reason about the transport.

The existing-config warning also says what it is warning about. It named the
enabled agents in a parenthetical — "already exists for this workspace (claude)" —
which read as though the workspace itself were called claude, and listing the
contents implied the config was per-agent when there is exactly one per workspace
covering agents, MCP servers, skills, tracing, and the budget policy. It now says
that scope plainly and itemizes nothing: an inventory doesn't change what the admin
should do, and `ucode setup show` prints the real thing for anyone comparing.

The duplicate "Fetching available models" spinner is gone. `discover_model_services`
and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a
run that configured Claude walked it twice. Rather than widen
`discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which
is what made this look expensive last time), the memo goes one layer down in
`list_model_services`, where both callers already converge — so no signature
changes and no stub churn. Cached per process and per workspace; failures are never
cached, so a transient error still retries; `use_cache=False` forces a fresh walk.
`conftest` clears it between tests so a cached listing can't leak into a test that
stubs the endpoint.

The Model Provider Service listing gets the same treatment, for the same reason: it
is workspace-wide and filtered per agent afterwards, so the wizard re-listed it once
per MPS-capable agent. One call now serves them all. Cached entries are copied on
both store and read, since the wizard treats the list as its own.

Its spinner is now conditional rather than removed. The cold listing takes ~1.4s, so
silence there would look like a hang; the cached ones are instant, and spinning
"Checking for model provider services for <agent>..." once per agent was what made
the wizard look like it re-listed them every time. The message no longer names an
agent either, since one lookup covers all of them.

Measured against eng-ml-inference.staging: 12 HTTP pages for the model catalog
across both callers, down from 24; and one MPS listing for claude + codex, down from
two.

Tests: +15. Mutation-verified — dropping either cache read fails a cache test,
dropping `searchable` fails the picker test, un-boxing the summary fails the panel
test, and re-adding the config inventory fails the warning test. The MPS aliasing
guard is split in two on purpose: the store-side and read-side copies each need
their own case, since a single test passed with either one still present.

Two more redundant lines removed while in here. The "Next steps" block no longer
suggests `ucode configure --dry-run`: the manifest describes what *developers*
should get, while that command previews *this machine's* agent configs, so it
implied a local test it does not perform. Publishing is the only next step there is
today. And the wizard's own `ensure_databricks_auth` is now quiet, because
`configure_shared_state` authenticates a moment later and prints the same "auth
already available" success — the wizard's call still has to run first, since the
admin gate and the existing-config check both need a token before discovery.

Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the
default agent against the authored config, so an admin can try it locally before
publishing. That needs both the flag and bare-`ucode` launch behavior, neither of
which exists yet.

Co-authored-by: Isaac
@tt-le
tt-le force-pushed the tien/managed-setup-wizard branch from 7f8ac63 to 30154f4 Compare August 5, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants