Skip to content

feat(desktop): expand a custom harness definition into per-profile variants - #7782

Open
majieddd wants to merge 1 commit into
block:mainfrom
majieddd:feat/harness-profile-variants
Open

majieddd wants to merge 1 commit into
block:mainfrom
majieddd:feat/harness-profile-variants

Conversation

@majieddd

Copy link
Copy Markdown

Implements #7781.

What this adds

A custom harness definition can now declare a variants block, and Buzz expands that one definition into one catalog entry per detected profile directory. Before this, a runtime with one command but several isolated homes (profiles, projects, per-agent config dirs) needed one hand-written definition file per home, kept in sync by hand: add or rename a profile and the runtime dropdown is wrong until someone regenerates those files.

{
  "id": "hermes-profiles",
  "label": "Hermes Agent",
  "command": "hermes-acp",
  "modelSelection": "harness",
  "variants": {
    "dir": "~/.hermes/profiles",
    "marker": "profile.yaml",
    "idTemplate": "{id}-{slug}",
    "labelTemplate": "{label} [{meta}] ({name})",
    "labelFrom": { "file": "profile.yaml", "key": "ui_meta.hermes-bots.title" },
    "env": { "HERMES_HOME": "{dir}" }
  }
}

That definition yields the template entry plus one generated entry per profile directory holding profile.yaml, each with its own HERMES_HOME, and each labelled from that profile's own persona title, for example Hermes Agent [Generalist] (generalist).

  • dir and marker: which directories count. A directory without the marker is skipped, so dir can point at a shared parent.
  • Templates: {id}, {label}, {name} (directory name), {slug} (id-safe directory name), {meta} from labelFrom, {root} (the scanned dir), {dir} (this profile's dir), usable in idTemplate, labelTemplate, args, and env.
  • labelFrom: one dotted key read out of a file inside the profile directory, with a fallback to a plain label when that file or key is missing, so an unreadable profile degrades instead of vanishing.
  • max: optional; expansion is capped at MAX_HARNESS_VARIANTS regardless.
  • A definition without variants returns exactly itself as one entry, so existing definitions are unaffected.

Two things that go with it

Generated entries are read-only. They carry generated plus generated_from, render with no edit affordance, and name the definition the user should edit instead (screenshot 1). A derived entry that can be edited is a derived entry that gets silently overwritten by the next scan, which is the failure mode this design is avoiding.

modelSelection: "harness". Optional, on any definition. When set, the shared agent-config renderer omits the model field and reports the named reason ownedByHarnessSelection, instead of offering a picker whose value the harness owns and ignores (screenshot 2). This is on the definition rather than keyed to a runtime id, so any harness that owns model choice can use it.

Implementation notes

  • desktop/src-tauri/src/managed_agents/custom_harnesses/variants.rs (new, 414 lines): expansion, {placeholder} rendering, labelFrom reading, validation.
  • custom_harnesses/variants/tests.rs (new, 328 lines): 12 unit tests, all driven through load_custom_harnesses after a real save_custom_harness_to_dir round-trip, so removing the expansion call in the loader, the variants validation, or the labelFrom read fails them.
  • Validation rejects a labelFrom file path that escapes the profile directory, an empty file or key, and env that the spawn path would reject anyway. Three of the twelve tests cover exactly that.
  • custom_harness_entry(...) is now the single constructor for catalog entries, shared by the command layer and the tests, instead of two places assembling them and drifting. commands/agent_discovery.rs loses 32 lines to it.
  • Generated ids are validated for collision the same way authored ones are, and the template keeps its own entry, so a user can still see and edit the definition that produced the family.

Manual test

  1. Put the definition above at <app_data_dir>/custom_harnesses/hermes-profiles.json (the same directory the Settings harness gallery writes to), adjusted to a profile root that exists on your machine.
  2. Open Settings, Harnesses. The definition row keeps its actions menu offering Edit and Delete; each generated profile row appears as Hermes Agent [Generalist] (generalist) with no actions menu and the note naming the definition to edit.
  3. Create an agent and pick a profile entry as the harness: the dialog shows Agent name, Description, Agent instructions, Agent harness, Advanced, and no model picker. Switch to Buzz Agent and the model picker comes back, so the omission follows the definition rather than the dialog.
  4. Add a directory to the profile root with the marker file and reopen the panel: a new entry appears with no rebuild.

Verification

All of the following ran at the pushed commit 4752836374a1fb6328562a54e3874c7659ec6481 (22 files, +1453/-84).

cargo test -p buzz-desktop --lib
  test result: ok. 3058 passed; 0 failed; 15 ignored; finished in 128.30s

npm run build:e2e                                   -> built in 18.97s, exit 0
playwright test tests/e2e/harness-profile-variants-screenshots.spec.ts --project=smoke
  ok 1  profile variants: generated entries are read-only and name their definition  (2.9s)
  ok 2  profile variants: a harness-owned model omits the agent model picker         (4.1s)
  2 passed (10.2s), exit 0
tsc --noEmit                                        -> exit 0
biome check .                                       -> exit 0
node ./scripts/check-file-sizes.mjs                 -> exit 0
node ./scripts/check-px-text.mjs                    -> exit 0
node ./scripts/check-pubkey-truncation.mjs          -> exit 0

The two spec frames are regenerated on every run and are byte-identical across runs (sha256 00f84233288c879fd59c4b4a78fced89a7deabfff5f4594729d02d5a6984c69c and 6ce5151a439391349f50ada72ebc4153d2569d0d1b9ed910f6a572c5be1c0660), so the images below come from this exact commit.

Two things I am not claiming as green, since neither is:

  1. biome check . reports 2 warnings and 5 infos, and none of the seven is in a file this commit touches: src/features/home/ui/inboxReopenNavigation.test.mjs (3, useTemplate), src/features/sidebar/lib/channelMutesStorage.test.mjs (useLiteralKeys), src/features/sidebar/lib/channelStarsStorage.test.mjs (useLiteralKeys), src/features/messages/ui/useAgentAddressLockPicker.ts (useOptionalChain), src/shared/styles/globals/terminal.css:276 (noImportantStyles). Exit code is 0, so this matches the state of main as far as I can tell, but it is not zero.
  2. The full desktop unit suite is 6514 pass / 1 fail: src/shared/lib/useDocumentVisible.test.mjs:254, focused polling pauses on blur and resumes after activation yields. It is intermittent on this tree (3 consecutive runs of that single file: fail, pass, pass) and this commit touches no file in that test's import graph, so I read it as a pre-existing flake rather than something introduced here. Both facts are stated so a maintainer can disagree with the read.

Toolchain substitutions, since this changes how much my numbers mean

  • The Hermit toolchain does not bootstrap on this Windows box (bin/hermit exits with /pkg/hermit@stable/hermit: No such file or directory), so just ci and just desktop-screenshot were replaced by the underlying commands run directly: npm run build:e2e plus Playwright for the frames, and cargo from rustup 1.95.0 at ~/.cargo/bin (OPUS_LIB_DIR=desktop/src-tauri/target/opus-prebuilt, CMAKE_GENERATOR=Ninja) for the Rust tests. These are not pinned-toolchain runs.
  • Playwright needed the pinned chromium headless shell (chromium_headless_shell-1223, about 112 MB from the Playwright CDN), since the revision cached here was older, and a python3 shim for the static server that playwright.config.ts starts with python3 -m http.server. Recipe written up in a local guide if useful.
  • cargo clippy locally is red for pre-existing reasons: 53 errors across 26 files, with an empty intersection with this diff's file set, measured earlier in this work on the same content. I did not re-run clippy at this commit, so treat CI's clippy as the authority here rather than anything I say about it.

Screenshots

The two frames are in the screenshot comment below, pinned to the same commit SHA as the checks above. They are hosted on a branch of my fork rather than through scripts/post-screenshots.sh, because that script pushes agent-screenshots/<user> to origin (block/buzz) and computes its raw URLs from block/buzz; as a contributor without write access I cannot push there, so I mirrored the script's file naming (pr-<n>--<name>.png) and SHA-pinned URL shape onto my fork. No relay media URLs are used, and scripts/check-pr-image-urls.sh passes on the same markdown.

Duplicate check

Searched open and closed PRs and issues for "harness profile", "custom harness variants", "harness definition profiles", "definitionVariants", "generated_from harness", "profile directory catalog", "variants block harness": no existing PR or issue for this primitive. Closest adjacent reports, all different from this change: #5813 (Edit Agent dialog: Harness field not editable), #5155 (custom ACP harnesses meeting managed-agents.json and the model dropdown), #7466 (starter profiles optional and removable).

Deferred, not in this PR

  • A panel action that rescans the profile root on demand instead of on panel open.
  • Surfacing labelFrom read failures in the UI; today an unreadable profile degrades to the plain label and is logged as a warning.
  • Per-variant logos: every generated entry resolves its icon the way the definition does, keyed through RUNTIME_LOGOS, so two profiles of one runtime cannot look different in the picker.

Attribution

Implemented by an AI agent (Hermes) running on the author's machine, at the author's direction; the author is the accountable author of this PR. AI-assisted submissions are explicitly welcome per CONTRIBUTING, and I am saying so rather than leaving it to be inferred.

…riants

A custom harness definition file can now declare a `variants` block. Buzz
expands that one definition into one catalog entry per detected profile
directory, so a runtime picker gains entries like
"Hermes Agent [Generalist] (generalist)" with no runtime-specific code
anywhere in the desktop shell.

- `variants` takes `dir`, `marker`, `idTemplate`, `labelTemplate`,
  `labelFrom`, `args` and `env`, expands `~/`, and is capped by
  `MAX_HARNESS_VARIANTS`.
- `labelFrom` reads one dotted key out of a file in the profile directory
  (e.g. `profile.yaml` -> `ui_meta.hermes-bots.title`) to fill `{meta}`.
- A definition without `variants` returns itself as one entry, unchanged; a
  definition with `variants` returns itself (the user-editable template)
  plus one generated entry per detected profile.
- Generated entries carry `generated` + `generated_from`, render read-only
  in the harness gallery, and name the definition the user should edit
  instead of offering an edit affordance the next scan would overwrite.
- `modelSelection: "harness"` flows to `AcpRuntimeCatalogEntry`, and the
  shared agent-config renderer omits the model field with the reason
  `ownedByHarnessSelection`, so the harness owns model choice.
- `custom_harness_entry(...)` becomes the single constructor for catalog
  entries, shared by the discovery command layer and the tests.

Tests: 12 unit tests in
`managed_agents/custom_harnesses/variants/tests.rs`, all driven through the
production seam (`load_custom_harnesses` / `save_custom_harness_to_dir`);
new browser spec
`desktop/tests/e2e/harness-profile-variants-screenshots.spec.ts`, registered
in the Playwright `smoke` project, asserts the read-only generated rows and
the omitted model picker and captures the two proof frames.

Signed-off-by: Majied LaFleur <bigmandad@users.noreply.github.com>
@majieddd
majieddd requested a review from a team as a code owner September 21, 2026 15:50
@github-actions

Copy link
Copy Markdown

🔐 Codex Security Review

Status: review required for the current range.

The current range is 5079c770fe30bb3d8204822ce6c2431eacac6d4b...4752836374a1fb6328562a54e3874c7659ec6481.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 4752836374a1fb6328562a54e3874c7659ec6481 to authorize a new review.
Any previous review applies only to its recorded range.

@majieddd

Copy link
Copy Markdown
Author

Both frames come from the pushed commit 4752836374a1fb6328562a54e3874c7659ec6481. The spec regenerates them on every run and they came back byte-identical, so what is below is this commit's output, not a stale capture.

1. Settings > Agent runtimes: generated entries are read-only

The definition row Hermes Agent (profiles) keeps its actions menu (Edit and Delete are still in it). The two expanded profiles render as Hermes Agent [Developer] (default) and Hermes Agent [Generalist] (generalist), each with no actions menu at all, and each carrying the note:

Generated from the "hermes-profiles" harness definition. Change that definition to change this entry.

The spec asserts the generated rows have no actions-menu trigger while the definition row's menu still offers Edit and Delete, so a regression that re-adds an edit affordance to a derived row fails it rather than passing quietly.

harnesses-generated-entries

2. Add agent: a harness-owned model omits the model picker

With a profile entry selected the dialog shows Agent name, Description, Agent instructions, Agent harness, Advanced. There is no model picker, no provider field and no AI-mode switch, and the picker returns when you switch to Buzz Agent, so the omission is caused by the definition's modelSelection: "harness" rather than by a fixture.

agent-no-model-picker


Where these are hosted. scripts/post-screenshots.sh pushes agent-screenshots/<user> to origin (block/buzz) and pins its raw URLs to block/buzz, which a contributor without write access to this repo cannot do. I mirrored that script's file naming (pr-<n>--<name>.png) and its commit-SHA-pinned URL shape onto my fork instead: branch agent-screenshots/majieddd at a59eccb, so the URLs above are immutable and stay valid for this PR. scripts/check-pr-image-urls.sh passes on this markdown, and both URLs return HTTP 200 image/png to an anonymous fetch, which is what GitHub's Camo proxy does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant