Add ucode setup: interactive managed-config authoring - #268
Open
tt-le wants to merge 3 commits into
Open
Conversation
tt-le
force-pushed
the
tien/managed-setup-serializer
branch
from
August 5, 2026 17:50
f4c989a to
d820408
Compare
The admin-facing wizard on top of the serializer added in the parent commit.
Walks a workspace admin through agents, per-agent models, tracing, MCP servers,
skills, and a spend-routing budget policy, then writes the manifest to
`~/.ucode/managed-settings.json`. Publishing is a separate `ucode apply` (next
change) so an admin can review the file first — and `ucode setup show` prints both
the summary and the exact proto-JSON that apply will POST.
`managed_wizard.py` is the interaction layer only; catalogs, validation, and
serialization stay in `managed_setup.py`. Sub-flows an admin already knows are
delegated to the existing commands (`configure_tracing_command`,
`configure_mcp_command`, `configure_skills_mcp_command`) and their results read back
out of `state.json`, so there is still exactly one picker per concern.
Claude Code is prompted per model family, not as a flat list. `ClaudeModelConfig.models`
is a `ClaudeDefaultModels` message — four optional slots (`default_opus_model` /
`default_sonnet_model` / `default_haiku_model` / `default_fable_model`), since Claude
Code addresses models by family alias rather than from a list, and a flat multi-select
cannot express that. Each family is skippable (the slots are `optional`, and an unset
one falls back to `default_model`); the overall `default_model` is then chosen from the
slots that were filled, so it can never name a model the config doesn't carry. With one
slot filled there is nothing to choose, so it reports what it inferred rather than
asking a single-option question.
That required looking past `state["claude_models"]`, which is `{family: newest}` because
the launch path pins one model per alias — the workspace tested against has 12 Claude
models and the bucketed view shows 4. `discover_claude_models_unbucketed` returns the
full `system.ai.claude-*` listing, `claude_family_candidates` groups it by family, and
`_known_models` reads the same set so validation cannot reject a version the prompts
offered. `discover_model_services` is untouched — every agent's config writer depends on
the shape of `claude_models`.
Provider-service agents get a real picker too. `list_model_provider_services` already
returns each service's `config.targets` (the provider-side model ids); the wizard now
selects the whole service dict and offers those, instead of an empty text prompt that
required knowing a provider-side id by heart. Two cases legitimately have nothing to
enumerate — `allow_all_targets`, and a relayed Anthropic subscription service that
routes by canonical name — and those fall back to free text, but say why and give an
example.
Gemini and Copilot get one model rather than a multi-select: their writers take a single
model and set one env var, so extra picks were silently discarded at apply time. This is
a wizard restriction, not a serializer one — both protos do declare `repeated string
models`, so `_FLAT_MODEL_LIST_AGENTS` is unchanged and a hand-written `--from-file`
manifest still serializes a list. OpenCode and Pi keep their multi-select; they really
do show a model picker.
Nothing is pre-checked in the model multi-selects. `options[0]` is whatever discovery
sorted first, not a recommendation — for Pi a Claude model, for Codex the oldest GPT —
so pre-checking it made "hit Enter" produce an arbitrary config and read as an
endorsement. It only existed to make Enter work at all, which stopped mattering once
every agent required a model: an empty selection is re-prompted
(`_require_selection` / `_require_multi_selection` / `_require_text`), which also makes
`_prompt_models_for_agent` total. Without that, picking a multi-model agent and hitting
Enter produced an agent with no `model_config`, and if it was the `default_agent` the run
died at the *end* — after tracing, MCP, skills, and budget had all been answered. Ctrl-C
still aborts.
Budget tiers offer only the models their agent was configured with, via
`configured_models_for_agent`. The picker previously read the workspace catalog and
called `load_state()`, so it could not have seen this run's choices even in principle —
configuring Pi with one Kimi model still offered every family. Picking one of the extras
was not harmless: `validateStoredConfig` checks only that a tier's `default_agent` is in
`enabled_agents`, never that the agent has the model, so the tier would activate and hand
a developer a model their agent was never given. `validate_manifest` gains that missing
rule, which is the durable half — it guards `--from-file` and hand-written manifests too.
Tiers are prompted in percent and stored as fractions (`prompt_for_percentage` converts),
keeping the spec-vs-API units mismatch in one place.
Model ids stay bare (`system.ai.claude-opus-4-8`), not provider-prefixed: each agent's
writer adds whatever its config format needs (see `opencode._resolve_model_selector`), so
the manifest stays agent-neutral and matches the spec's examples.
Two readback details worth noting:
- Tracing is stored as `tracing.uc_destination` but the managed config calls it
`tracing.table`; an enabled-but-not-UC-backed experiment has no table, so it is
omitted rather than published empty.
- `state.json` records each MCP server's resolved URL but not its type, while the
manifest needs `{name, type}`. `_mcp_type_for_url` maps the URL shape back to a type
and skips anything unrecognized instead of guessing.
`databricks.py`: `is_workspace_admin` (SCIM `Me` group membership) and
`list_workspace_budgets` (workspace-scoped, so no account auth is needed — ucode never
creates budgets, an admin picks an existing one). The admin gate errors on a definite
non-admin and warns-then-continues when the check itself fails, since the API enforces
the same rule server-side.
`ui.py`: `preselected`/`prompt` on `prompt_for_tools`, plus `prompt_for_multi_selection`,
`prompt_for_text`, and `prompt_for_percentage`. Reuses the existing
`prompt_yes_no_default` rather than adding a second defaulting yes/no.
README gains a "Managed config for a workspace (admins)" section, `setup` rows in Other
Commands, and `~/.ucode/managed-settings.json` in Managed Local Files. It states plainly
that the wizard leaves your agent configs alone *except* when you accept the tracing /
MCP / skills sub-steps, which do configure this machine — the one thing about the flow
that isn't obvious.
Verified interactively against eng-ml-inference.staging: six opus versions and four
sonnet offered, a skipped family absent from `models`, and the result round-trips through
`normalize_managed_config` to the proto shape.
Tests: 116 cases. Several are mutation-checked — reverting the claude flat multi-select
fails eight, reverting the budget-tier picker fails
`test_offers_only_the_models_the_agent_was_configured_with`, and dropping the tier-model
validation rule fails `test_tier_model_must_be_one_the_agent_has`. Note `typer.Exit`
subclasses RuntimeError, so raising it inside the command's try block made a successful
run print "ERROR 0"; fixed by exiting after the handler, with
`test_successful_setup_exits_zero` failing if that regresses.
Known gap: `discover_model_services` and `discover_claude_models_unbucketed` both page
`/api/2.1/unity-catalog/model-services`, so a run that configures Claude lists it twice.
`_claude_candidates` caches on `state["all_claude_models"]`, which bounds it to once per
run, but removing the second listing means widening `discover_model_services`' 5-tuple
return — asserted in eight tests and mocked in sixteen — so it is left for a follow-up.
Co-authored-by: Isaac
Two CI failures, both passing locally for environment reasons. `_claude_candidates` guarded its catalog fetch with `except RuntimeError`, but `get_databricks_token` shells out to `databricks auth token` — so a machine without the CLI on PATH raises `FileNotFoundError`, which is an `OSError` and sailed straight through. This is a real bug, not just a test artifact: any user without the Databricks CLI installed hit an uncaught traceback mid-wizard instead of degrading to the bucketed per-family picks. Now catches `OSError` too. `test_claude_candidates_survive_a_missing_databricks_cli` covers it, and `test_every_agent_always_gets_a_default_model` — which reached the network on its claude pass — now stubs the fetch rather than depending on the host. `test_setup_help_lists_from_file` grepped `--from-file` out of rendered `--help` output. Rich ellipsizes option names to fit the terminal, so below roughly 40 columns it renders `--fro…`; hosted runners report no width and get the narrow fallback. It now asserts on the declared Click option, which no amount of wrapping can change, and still checks the help renders successfully. Verified by reverting each fix in turn: the first fails with the same `FileNotFoundError: 'databricks'` CI reported, the second with the same missing `--from-file` assertion at `COLUMNS=30`. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 17:53
48d9066 to
b078138
Compare
AarushiShah-db
requested changes
Aug 5, 2026
Collaborator
There was a problem hiding this comment.
Some high level comments based on the video
- Configuration Summary - can we put a box around it
- Dry run: is there a reason why its ucode configure --dry-run? Can it just be ucode —dry-run?
- Warning message update: "A published Claude configuration already exists for this workspace. Publishing will replace it. Make sure this configuration includes everything you want to keep."
- For the model picker , can we add option to filter by search (you type and it will filter the models) Same with budget policy tracker
- What is the initial "Fetching available models" step doing and is that needed? because aren't we doing individual API calls anyways to get the models for the individual agents?
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together, down from 24. Tests: +9. Mutation-verified — dropping the cache read fails two cache tests, dropping `searchable` fails the picker test, and un-boxing the summary fails the panel test. Deferred, with Aarushi's agreement: the dry-run hint still points at `ucode configure --dry-run`. The intent is a root `ucode --dry-run` that launches the default agent against the authored managed config, which needs both that flag and bare-`ucode` launch behavior that doesn't exist yet. Co-authored-by: Isaac
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together, down from 24. Tests: +9. Mutation-verified — dropping the cache read fails two cache tests, dropping `searchable` fails the picker test, and un-boxing the summary fails the panel test. Deferred, with Aarushi's agreement: the dry-run hint still points at `ucode configure --dry-run`. The intent is a root `ucode --dry-run` that launches the default agent against the authored managed config, which needs both that flag and bare-`ucode` launch behavior that doesn't exist yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 20:49
7fff00b to
b8e191a
Compare
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together, down from 24. Tests: +9. Mutation-verified — dropping the cache read fails two cache tests, dropping `searchable` fails the picker test, and un-boxing the summary fails the panel test. Two more redundant lines removed while in here. The "Next steps" block no longer suggests `ucode configure --dry-run`: the manifest describes what *developers* should get, while that command previews *this machine's* agent configs, so it implied a local test it does not perform. Publishing is the only next step there is today. And the wizard's own `ensure_databricks_auth` is now quiet, because `configure_shared_state` authenticates a moment later and prints the same "auth already available" success — the wizard's call still has to run first, since the admin gate and the existing-config check both need a token before discovery. Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the default agent against the authored config, so an admin can try it locally before publishing. That needs both the flag and bare-`ucode` launch behavior, neither of which exists yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 21:03
b8e191a to
57cb08d
Compare
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together, down from 24. Tests: +9. Mutation-verified — dropping the cache read fails two cache tests, dropping `searchable` fails the picker test, and un-boxing the summary fails the panel test. Two more redundant lines removed while in here. The "Next steps" block no longer suggests `ucode configure --dry-run`: the manifest describes what *developers* should get, while that command previews *this machine's* agent configs, so it implied a local test it does not perform. Publishing is the only next step there is today. And the wizard's own `ensure_databricks_auth` is now quiet, because `configure_shared_state` authenticates a moment later and prints the same "auth already available" success — the wizard's call still has to run first, since the admin gate and the existing-config check both need a token before discovery. Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the default agent against the authored config, so an admin can try it locally before publishing. That needs both the flag and bare-`ucode` launch behavior, neither of which exists yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 21:12
57cb08d to
c536a37
Compare
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. Measured against eng-ml-inference.staging: 12 HTTP pages for both callers together, down from 24. Tests: +9. Mutation-verified — dropping the cache read fails two cache tests, dropping `searchable` fails the picker test, and un-boxing the summary fails the panel test. Two more redundant lines removed while in here. The "Next steps" block no longer suggests `ucode configure --dry-run`: the manifest describes what *developers* should get, while that command previews *this machine's* agent configs, so it implied a local test it does not perform. Publishing is the only next step there is today. And the wizard's own `ensure_databricks_auth` is now quiet, because `configure_shared_state` authenticates a moment later and prints the same "auth already available" success — the wizard's call still has to run first, since the admin gate and the existing-config check both need a token before discovery. Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the default agent against the authored config, so an admin can try it locally before publishing. That needs both the flag and bare-`ucode` launch behavior, neither of which exists yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 21:21
c536a37 to
0d02f98
Compare
tt-le
added a commit
that referenced
this pull request
Aug 5, 2026
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The existing-config warning also says what it is warning about. It named the enabled agents in a parenthetical — "already exists for this workspace (claude)" — which read as though the workspace itself were called claude, and listing the contents implied the config was per-agent when there is exactly one per workspace covering agents, MCP servers, skills, tracing, and the budget policy. It now says that scope plainly and itemizes nothing: an inventory doesn't change what the admin should do, and `ucode setup show` prints the real thing for anyone comparing. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. The Model Provider Service listing gets the same treatment, for the same reason: it is workspace-wide and filtered per agent afterwards, so the wizard re-listed it once per MPS-capable agent. One call now serves them all. Cached entries are copied on both store and read, since the wizard treats the list as its own. Measured against eng-ml-inference.staging: 12 HTTP pages for the model catalog across both callers, down from 24; and one MPS listing for claude + codex, down from two. Tests: +15. Mutation-verified — dropping either cache read fails a cache test, dropping `searchable` fails the picker test, un-boxing the summary fails the panel test, and re-adding the config inventory fails the warning test. The MPS aliasing guard is split in two on purpose: the store-side and read-side copies each need their own case, since a single test passed with either one still present. Two more redundant lines removed while in here. The "Next steps" block no longer suggests `ucode configure --dry-run`: the manifest describes what *developers* should get, while that command previews *this machine's* agent configs, so it implied a local test it does not perform. Publishing is the only next step there is today. And the wizard's own `ensure_databricks_auth` is now quiet, because `configure_shared_state` authenticates a moment later and prints the same "auth already available" success — the wizard's call still has to run first, since the admin gate and the existing-config check both need a token before discovery. Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the default agent against the authored config, so an admin can try it locally before publishing. That needs both the flag and bare-`ucode` launch behavior, neither of which exists yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 22:13
0d02f98 to
7f8ac63
Compare
Review feedback on #268. The configuration summary is boxed. It is the one block an admin is meant to read as a whole and check against what they intended, and it arrives after a long flow of prompts, where loose key/value lines blend into everything printed before them. `ui.print_panel` boxes a body (as opposed to `print_section`, which boxes a bare title) and `ui.kv_line` returns a `print_kv`-styled line for collecting into one. Every picker in the flow now filters as you type. On the workspace this was built against, the model lists run to 16 GPT entries and 12 Claude ones, which is more than is comfortable to arrow through. questionary supports this directly, with one constraint worth naming: it refuses `use_search_filter` together with `use_jk_keys` (j and k are search characters), so search costs j/k navigation. Arrow keys still work, and it is opt-in via a `searchable` flag so short pickers — "Databricks Hosted vs External Models" — keep j/k. The existing-config warning now says what actually happens. It claimed "there is no partial update yet", which was both vague and, once `ucode apply` PATCHes, no longer true. Publishing is a full replace of the workspace's config, so the warning says that plainly and names the agents already published, which is what the admin stands to lose. Field masks and PATCH stay out of it — an admin doesn't need to reason about the transport. The existing-config warning also says what it is warning about. It named the enabled agents in a parenthetical — "already exists for this workspace (claude)" — which read as though the workspace itself were called claude, and listing the contents implied the config was per-agent when there is exactly one per workspace covering agents, MCP servers, skills, tracing, and the budget policy. It now says that scope plainly and itemizes nothing: an inventory doesn't change what the admin should do, and `ucode setup show` prints the real thing for anyone comparing. The duplicate "Fetching available models" spinner is gone. `discover_model_services` and `discover_claude_models_unbucketed` both page the whole metastore catalog, so a run that configured Claude walked it twice. Rather than widen `discover_model_services`' 5-tuple return (asserted in 8 tests, mocked in 19, which is what made this look expensive last time), the memo goes one layer down in `list_model_services`, where both callers already converge — so no signature changes and no stub churn. Cached per process and per workspace; failures are never cached, so a transient error still retries; `use_cache=False` forces a fresh walk. `conftest` clears it between tests so a cached listing can't leak into a test that stubs the endpoint. The Model Provider Service listing gets the same treatment, for the same reason: it is workspace-wide and filtered per agent afterwards, so the wizard re-listed it once per MPS-capable agent. One call now serves them all. Cached entries are copied on both store and read, since the wizard treats the list as its own. Its spinner is now conditional rather than removed. The cold listing takes ~1.4s, so silence there would look like a hang; the cached ones are instant, and spinning "Checking for model provider services for <agent>..." once per agent was what made the wizard look like it re-listed them every time. The message no longer names an agent either, since one lookup covers all of them. Measured against eng-ml-inference.staging: 12 HTTP pages for the model catalog across both callers, down from 24; and one MPS listing for claude + codex, down from two. Tests: +15. Mutation-verified — dropping either cache read fails a cache test, dropping `searchable` fails the picker test, un-boxing the summary fails the panel test, and re-adding the config inventory fails the warning test. The MPS aliasing guard is split in two on purpose: the store-side and read-side copies each need their own case, since a single test passed with either one still present. Two more redundant lines removed while in here. The "Next steps" block no longer suggests `ucode configure --dry-run`: the manifest describes what *developers* should get, while that command previews *this machine's* agent configs, so it implied a local test it does not perform. Publishing is the only next step there is today. And the wizard's own `ensure_databricks_auth` is now quiet, because `configure_shared_state` authenticates a moment later and prints the same "auth already available" success — the wizard's call still has to run first, since the admin gate and the existing-config check both need a token before discovery. Deferred, with Aarushi's agreement: a root `ucode --dry-run` that launches the default agent against the authored config, so an admin can try it locally before publishing. That needs both the flag and bare-`ucode` launch behavior, neither of which exists yet. Co-authored-by: Isaac
tt-le
force-pushed
the
tien/managed-setup-wizard
branch
from
August 5, 2026 23:06
7f8ac63 to
30154f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Demo
Screen.Recording.2026-08-05.at.10.11.05.AM.mov
Changes
The admin-facing wizard on top of #267's serializer. Walks a workspace admin through agents, per-agent models, tracing, MCP servers, skills, and a spend-routing budget policy, then writes the manifest to
~/.ucode/managed-settings.json. Publishing is a separateucode apply(next PR) so an admin can review the file first — anducode setup showprints both the summary and the exact proto-JSON that apply will POST.managed_wizard.pyis the interaction layer only; catalogs, validation, and serialization stay inmanaged_setup.py. Sub-flows an admin already knows are delegated to the existing commands (configure_tracing_command,configure_mcp_command,configure_skills_mcp_command) and their results read back out ofstate.json, so there is still exactly one picker per concern.Claude Code is prompted per model family
ClaudeModelConfig.modelsis aClaudeDefaultModelsmessage — four optional slots (default_opus_model/default_sonnet_model/default_haiku_model/default_fable_model), since Claude Code addresses models by family alias rather than from a list. A flat multi-select cannot express that.Each family is skippable (the slots are
optional, and an unset one falls back todefault_model); the overalldefault_modelis then chosen from the slots that were filled, so it can never name a model the config doesn't carry. With one slot filled there is nothing to choose, so it reports what it inferred rather than asking a single-option question.That required looking past
state["claude_models"], which is{family: newest}because the launch path pins one model per alias — the workspace tested against has 12 Claude models and the bucketed view shows 4.discover_claude_models_unbucketedreturns the full listing,claude_family_candidatesgroups it by family, and_known_modelsreads the same set so validation cannot reject a version the prompts offered.discover_model_servicesis untouched — every agent's config writer depends on the shape ofclaude_models.Other model-prompt behaviour
— Provider-service agents get a real picker.
list_model_provider_servicesalready returns each service'sconfig.targets; the wizard now selects the whole service dict and offers those, instead of an empty text prompt that required knowing a provider-side id by heart.allow_all_targetsand relayed Anthropic subscription services have nothing to enumerate, so those fall back to free text — but say why and give an example.— Gemini and Copilot get one model, not a multi-select: their writers take a single model and set one env var, so extra picks were silently discarded at apply time. Wizard-level only — both protos declare
repeated string models, so--from-filestill serializes a list. OpenCode and Pi keep their multi-select; they really do show a model picker.— Nothing is pre-checked.
options[0]is whatever discovery sorted first, not a recommendation — for Pi a Claude model, for Codex the oldest GPT. It only existed to make Enter work, which stopped mattering once every agent required a model: an empty selection is re-prompted. That also makes_prompt_models_for_agenttotal. Without it, picking a multi-model agent and hitting Enter produced an agent with nomodel_config, and if it was thedefault_agentthe run died at the end — after tracing, MCP, skills, and budget had all been answered.Budget tiers offer only the agent's own models
The picker previously read the workspace catalog and called
load_state(), so it could not have seen this run's choices even in principle — configuring Pi with one Kimi model still offered every family. Picking one of the extras was not harmless:validateStoredConfigchecks only that a tier'sdefault_agentis inenabled_agents, never that the agent has the model, so the tier would activate and hand a developer a model their agent was never given.validate_manifestgains that missing rule, which is the durable half — it guards--from-fileand hand-written manifests too.Tiers are prompted in percent and stored as fractions, keeping the spec-vs-API units mismatch in one place.
Smaller notes
Model ids stay bare (
system.ai.claude-opus-4-8), not provider-prefixed: each agent's writer adds whatever its config format needs, so the manifest stays agent-neutral.Two readback details: tracing is stored as
tracing.uc_destinationbut the managed config calls ittracing.table, and an enabled-but-not-UC-backed experiment has no table so it is omitted rather than published empty;state.jsonrecords each MCP server's URL but not its type, so_mcp_type_for_urlmaps the URL shape back and skips anything unrecognized instead of guessing.databricks.pyaddsis_workspace_admin(SCIMMegroup membership) andlist_workspace_budgets(workspace-scoped — ucode never creates budgets, an admin picks an existing one). The admin gate errors on a definite non-admin and warns-then-continues when the check itself fails, since the API enforces the same rule server-side.Testing
uv run pytest— 1334 passed, 36 skipped. 116 new cases. (tests/test_e2e_user_agent.py::test_user_agent_arrives_at_gatewayfails on my machine because/etc/claude-code/managed-settings.jsonoverridesANTHROPIC_BASE_URLpast the test's capture server; it fails onmaintoo and is unrelated.)Several are mutation-checked: reverting the claude flat multi-select fails eight, reverting the budget-tier picker fails
test_offers_only_the_models_the_agent_was_configured_with, and dropping the tier-model validation rule failstest_tier_model_must_be_one_the_agent_has.Verified interactively against a staging workspace: six opus versions and four sonnet offered, a skipped family absent from
models, and the result round-trips throughnormalize_managed_configto the proto shape.Known gap
discover_model_servicesanddiscover_claude_models_unbucketedboth page/api/2.1/unity-catalog/model-services, so a run that configures Claude lists it twice (one extra spinner)._claude_candidatescaches onstate["all_claude_models"], bounding it to once per run. Removing the second listing means wideningdiscover_model_services' 5-tuple return, which is asserted in eight tests and mocked in sixteen — left for a follow-up rather than growing this PR.This pull request and its description were written by Isaac.