TypeAgent Studio: Impact Report redesign + connection UX#2553
Open
TalZaccai wants to merge 23 commits into
Open
TypeAgent Studio: Impact Report redesign + connection UX#2553TalZaccai wants to merge 23 commits into
TalZaccai wants to merge 23 commits into
Conversation
…atcher, replay Add computeActionSchemaFileHash to agent-cache (schemaInfoProvider) as the single source of truth for the schema cache-namespace hash. The dispatcher's actionSchemaFileCache and typeagent-core's constructionCacheResolver now both delegate to it, removing two copies of the sha256/base64 hashing logic (and a copy-pasted helper in the dispatcher test). Aligns the config-truthiness check so the namespace key is byte-identical across producers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Promote the crypto-strong webview nonce generator to a new @typeagent/core/webview subpath. typeagent-studio's webviewKit re-exports it, and vscode-shell's chatViewProvider now uses it in place of a weak Math.random getNonce. CSP/HTML stay per-package (the chat UI needs 'unsafe-inline' styles; studio uses a style nonce) - only the nonce is shared. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a single createWebSocketRpcChannel to @typeagent/websocket-utils/rpcChannel (built on agent-rpc createChannelAdapter). The studio-service server and the studio extension client both re-export it, deleting two identical hand-rolled WebSocket-to-RpcChannel adapters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract resolveAgentName into a dependency-free sandbox/agentRef module. inMemorySandboxManager's default loader now uses it instead of a private deriveAgentName subset (which mishandled packages/agents/<name> paths); repoAgentLoader re-exports it so its API and tests are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the leftover 'Hello (skeleton)' placeholder command (and its activation event) and rename the connect command from the misleading 'Connect Event Log to studio service' to 'Connect to Studio service', since it connects every channel-backed view, not just the Event Log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tate to the Impact Report The report now opens with a differences-only filter (equal rows hidden) so a replay surfaces regressions instead of burying them under unchanged rows. Clickable per-status chips show live counts and toggle each status in place; a note reports how many rows are hidden, and an all-equal run shows a positive 'no differences' message instead of an empty table. Before any replay runs, a first-run empty state explains the A/B compare and how to start one. Filter and empty-state logic live as pure helpers in the view model (unit-tested); the client only does DOM wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ake empty chips inert The single crammed detail string is broken into discrete columns (Utterance, Status, Base (A), Compare (B), Latency) so each side's resolution and the latency pair are read at a glance instead of parsed out of one cell. A status filter chip with a zero count is now disabled rather than clickable, since there is nothing to filter on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both the Studio service connection and the vscode-shell agent-server bridge previously hand-rolled their own reconnect delays (an array clamped at its last value, and a linear ramp). Extract a single exponential-with-cap strategy into @typeagent/websocket-utils/backoff and wire both call sites to it, keeping each surface's local UI (Studio's single-flight retry timer and fencing; the shell's countdown ribbon and reconnectStatus broadcast). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The four VS Code tree views (Event Log, Collisions, Sandboxes, Corpora) each re-declared the same change emitter, connected flag + setConnected gate, refresh, and descriptor-to-TreeItem field mapping. Hoist those into BaseStudioTreeProvider<TNode>; each provider now supplies only what differs (getChildren, collapsible state, icon/command decoration) and overrides dispose to tear down its own subscription before calling super. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The event-log, replay, and corpus presenters each hand-rolled the same collapse-whitespace + cap-with-ellipsis routine. Extract collapseAndTruncate into textFormatting and delegate to it, so the truncation behaviour is defined once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The feedback-badge test used rating negative, which is not a member of FeedbackRating (up | down), so the file failed tsc type-checking even though esbuild/tsx run it regardless. Use down. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rework the Impact Report webview layout and styling, and add an All filter pill that shows every result and is selected by default after a run. Also share the collapse-and-truncate text helper in the replay view model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the Impact Report's version and agent selection as native VS Code QuickPicks driven entirely by the extension host (the webview never shells out to git — the security boundary). The host enumerates refs with two parallel git calls (one log for HEAD + current branch + recent commits, one for-each-ref for branches and tags), caches them per panel, and lazily appends remote-tracking branches on request. Users can also type a free-form commit SHA, tag, or relative ref (e.g. HEAD~3), validated on the host. Add row drill-in (U4a): clicking a changed/new/lost row opens a detail pane with a unified A/B JSON diff of the action. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Launch the Impact Report from a graph icon on each Corpora agent row instead of a single global panel + in-webview agent picker. Panels are now keyed per agent (WebviewKitPanel instanceKey), so reports for different agents open as separate tabs that can sit side-by-side; the agent is shown read-only in the action bar and in the tab title. The webview protocol drops the agent picker round-trip (init now carries the fixed agent + availability) and the host uses its authoritative agent for the replay. Also lands the native VS Code visual redesign of the report (codicon font wired through esbuild, action bar, native list/detail styling). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… indicator Collapse the four repeated 'not connected' welcome blocks into concise, button-less messages that defer to the status bar, and reframe the status bar's disconnected state as 'reconnecting' to reflect the existing auto-retry. In the Impact Report webview, drop the manual reconnect button for a single connection pill that mirrors the shared service connection and re-inits automatically on each reconnect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ect countdown Replace the per-view 'connecting' welcome blocks with a single global signal: the status bar item now shows a live 'reconnecting in Ns...' countdown, and each sidebar tree holds VS Code's native loading bar until the service connects (shared whenConnected gate in BaseStudioTreeProvider). Once connected, views render their real rows or empty-state placeholder rows. Removes the manual reconnect affordances and the duplicated welcome text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…he Impact Report Each table cell now carries a hover description: status terms (equal/changed/ new match/lost match), per-side resolution tokens (hit, miss, needs-explanation, llm-resolved, etc.), and the A/B latency pair. The latency column is left-aligned. Equal rows are now clickable too — they highlight as selected and close any open A/B diff pane (selection is tracked separately from the open detail so equal rows can read as selected with no pane). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icons for status Sandbox, Corpora, Event Log, and Collisions rows now build a structured, vscode-free TooltipModel that BaseStudioTreeProvider renders into a MarkdownString hover card (bold labels, code-styled hashes/paths/ids, an optional affordance hint). Status is carried by row icons rather than duplicated text: sandbox agents drop the redundant health word (keeping the schema fingerprint), corpus entries show a coloured thumbs-up/down icon instead of a feedback text badge, and collision exemplar rows drop their redundant description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Free-form and webview-supplied git refs were passed to git as positional arguments ahead of the \--\ path separator. git still parses a leading-dash token as an option there, so a ref like \--output=<path>\ was honored and could make \git log\/\git show\ write a file. Insert \--end-of-options\ before the ref in resolveRef, resolveVersionProvenance, and the replay grammar resolver's \git show <ref>:<path>\ so a leading-dash ref is always treated as a revision. Add a regression test asserting the guard precedes the ref. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
resolveAgentName derived the agent's package name from a hard-coded \packages/agents/\ marker, so a reference under a configured (non-default) agent root whose path ends in e.g. \<root>/<agent>/src\ would resolve to the wrong segment. Accept the configured agentRoots and, when the reference sits under one, take the first segment after the longest matching root; the marker remains a fallback for references that don't match a known root. repoAgentLoader now passes its resolved roots through. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Redesigns TypeAgent Studio’s “regression hunting” workflow by rebuilding the Impact Report as a VS Code-native per-agent webview, and unifies connection UX across Studio views via shared auto-reconnect/backoff infrastructure and shared UI helpers.
Changes:
- Rebuilds the Impact Report into a per-agent webview with native version pickers, status filter chips, drill-in action diffs, and run provenance capture.
- Introduces shared connection/backoff + RPC-channel helpers and refactors Studio tree providers to use a common base (native loading bar gating, Markdown hover cards, icon-driven status).
- Dedupes/centralizes webview nonce generation and schema-hash computation used across dispatcher/cache/replay tooling.
Show a summary per file
| File | Description |
|---|---|
| ts/pnpm-lock.yaml | Lockfile updates for new/shifted workspace deps (codicons, websocket-utils, agent-rpc). |
| ts/packages/vscode-shell/src/chatViewProvider.ts | Switches webview nonce generation to shared core helper. |
| ts/packages/vscode-shell/src/agentServerBridge.ts | Replaces ad-hoc reconnect backoff with shared exponential backoff utility. |
| ts/packages/vscode-shell/package.json | Adds dependency on shared @typeagent/websocket-utils. |
| ts/packages/utils/webSocketUtils/src/rpcChannel.ts | Adds shared adapter from ws WebSocket to agent-rpc channel abstraction. |
| ts/packages/utils/webSocketUtils/src/backoff.ts | Adds reusable exponential backoff helper with reset/attempt tracking. |
| ts/packages/utils/webSocketUtils/package.json | Exposes new backoff/rpcChannel subpath exports; adds agent-rpc dependency. |
| ts/packages/typeagent-studio/src/webviewKit/webviewHtml.ts | Removes local nonce implementation; re-exports shared nonce helper. |
| ts/packages/typeagent-studio/src/webviewKit/replayViewModel.ts | Expands Impact Report view model: filter chips, provenance formatting, action JSON diffing, and version-spec coercion. |
| ts/packages/typeagent-studio/src/webviewKit/protocol.ts | Updates host↔webview protocol: typed version specs, per-agent init, connection state mirroring, version picker messaging. |
| ts/packages/typeagent-studio/src/webviewKit/host.ts | Allows multiple webview panels per view type via instanceKey registry keying. |
| ts/packages/typeagent-studio/src/webviewKit/client/impactReport.ts | Major Impact Report client UI redesign (action bar, filters, drill-in diff pane, provenance, connection pill). |
| ts/packages/typeagent-studio/src/tooltipModel.ts | Adds vscode-free tooltip model shared by tree presenters for testable hover content. |
| ts/packages/typeagent-studio/src/textFormatting.ts | Centralizes whitespace collapsing + truncation used by trees/webviews. |
| ts/packages/typeagent-studio/src/test/webviewProtocol.spec.ts | Updates protocol parsing tests for new message shapes and stricter validation. |
| ts/packages/typeagent-studio/src/test/studioServiceConnection.spec.ts | Updates tests for new backoff options (baseBackoffMs). |
| ts/packages/typeagent-studio/src/test/sandboxTreePresentation.spec.ts | Updates expectations for icon-driven health and new tooltip model usage. |
| ts/packages/typeagent-studio/src/test/sandboxSource.spec.ts | Updates connection backoff option usage in tests. |
| ts/packages/typeagent-studio/src/test/replayViewModel.spec.ts | Adds/updates tests for new view-model helpers (filters, provenance, stable diffing). |
| ts/packages/typeagent-studio/src/test/gitRefProvider.spec.ts | Adds tests for host-side git ref enumeration, resolution, and --end-of-options hardening. |
| ts/packages/typeagent-studio/src/test/eventLogSource.spec.ts | Updates connection backoff option usage in tests. |
| ts/packages/typeagent-studio/src/test/corpusTreePresentation.spec.ts | Updates expectations for icon-driven feedback ratings and tooltip model. |
| ts/packages/typeagent-studio/src/test/collisionsSource.spec.ts | Updates connection backoff option usage in tests. |
| ts/packages/typeagent-studio/src/test/collisionsPresentation.spec.ts | Updates collision tooltip assertions to use structured tooltip model. |
| ts/packages/typeagent-studio/src/test/backoff.spec.ts | Adds unit tests for shared exponential backoff behavior. |
| ts/packages/typeagent-studio/src/studioServiceConnection.ts | Replaces manual backoff array with shared exponential backoff + exposes next retry time for countdown UI. |
| ts/packages/typeagent-studio/src/studioServiceClient.ts | Switches to shared WebSocket→RpcChannel adapter and re-exports it. |
| ts/packages/typeagent-studio/src/sandboxTreeProvider.ts | Refactors sandbox tree provider to extend shared base provider and gate loading on connection. |
| ts/packages/typeagent-studio/src/sandboxTreePresentation.ts | Moves tooltips to structured model and shifts health display to icons. |
| ts/packages/typeagent-studio/src/replayPresentation.ts | Uses shared text formatting helper for truncation. |
| ts/packages/typeagent-studio/src/impactReportView.ts | Reworks report hosting: per-agent panels, host-side git pickers, provenance capture, connection mirroring. |
| ts/packages/typeagent-studio/src/gitRefProvider.ts | Implements host-side git ref listing/resolution and provenance pinning with --end-of-options hardening. |
| ts/packages/typeagent-studio/src/extension.ts | Unifies connection UX via status bar countdown; opens Impact Report per agent; removes skeleton “hello” command. |
| ts/packages/typeagent-studio/src/eventLogTreeProvider.ts | Refactors to shared base tree provider and connection-gated loading behavior. |
| ts/packages/typeagent-studio/src/eventLogPresentation.ts | Adopts structured tooltip model and shared truncation helper. |
| ts/packages/typeagent-studio/src/corpusTreeProvider.ts | Refactors to shared base tree provider; adds icon mapping for feedback ratings. |
| ts/packages/typeagent-studio/src/corpusTreePresentation.ts | Moves entry tooltips to structured model; shifts feedback to icons; uses shared truncation helper. |
| ts/packages/typeagent-studio/src/collisionsTreeProvider.ts | Refactors to shared base tree provider and connection-gated loading behavior. |
| ts/packages/typeagent-studio/src/collisionsPresentation.ts | Converts collision/skipped/participant tooltips to structured model; minor row formatting changes. |
| ts/packages/typeagent-studio/src/baseTreeProvider.ts | New shared base tree provider: descriptor→TreeItem mapping, connection gating, Markdown tooltip rendering. |
| ts/packages/typeagent-studio/README.md | Removes obsolete “Hello (skeleton)” command mention. |
| ts/packages/typeagent-studio/package.json | Updates command titles/contributions; adds codicons dev dependency; removes viewsWelcome blocks and skeleton command. |
| ts/packages/typeagent-studio/media/impactReport.css | New Impact Report webview styling, including codicon font embedding and redesigned layout. |
| ts/packages/typeagent-studio/esbuild.mjs | Copies codicon.ttf into extension media assets at build time. |
| ts/packages/typeagent-studio/.gitignore | Ignores generated codicon.ttf asset. |
| ts/packages/typeagent-core/test/repoAgentLoader.spec.ts | Adds tests for agent-name resolution relative to configured agent roots. |
| ts/packages/typeagent-core/src/webview/index.ts | Adds shared, crypto-strong webview nonce generator. |
| ts/packages/typeagent-core/src/sandbox/repoAgentLoader.ts | Refactors agent-name resolution to shared helper and supports configured roots. |
| ts/packages/typeagent-core/src/sandbox/inMemorySandboxManager.ts | Uses shared agent-name resolver for default loader behavior. |
| ts/packages/typeagent-core/src/sandbox/agentRef.ts | New shared agent reference → agent name resolver (supports configured roots). |
| ts/packages/typeagent-core/src/replay/grammarResolver.ts | Hardens git show calls with --end-of-options to prevent option injection via refs. |
| ts/packages/typeagent-core/src/replay/constructionCacheResolver.ts | Delegates schema hash computation to shared cache helper to avoid drift. |
| ts/packages/typeagent-core/src/index.ts | Exports new webview submodule from core package. |
| ts/packages/typeagent-core/package.json | Adds ./webview export. |
| ts/packages/studio-service/src/studioServiceServer.ts | Switches to shared WebSocket→RpcChannel adapter and re-exports it. |
| ts/packages/dispatcher/dispatcher/test/actionSchemaFileCache.spec.ts | Updates tests to use shared schema-hash helper. |
| ts/packages/dispatcher/dispatcher/src/translation/actionSchemaFileCache.ts | Replaces local hashing with shared computeActionSchemaFileHash. |
| ts/packages/cache/src/index.ts | Exports shared schema-hash helper from cache package. |
| ts/packages/cache/src/explanation/schemaInfoProvider.ts | Implements shared computeActionSchemaFileHash used by dispatcher and consumers. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Files not reviewed (1)
- ts/pnpm-lock.yaml: Generated file
- Files reviewed: 58/59 changed files
- Comments generated: 3
CodeQL flagged \/\\/+\$/\ in resolveAgentName as a polynomial regex that can run slowly on a root string with many trailing slashes. Replace it with a linear character-walk helper (stripTrailingSlashes), mirroring the loop already used for the reference itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- createWebSocketRpcChannel: decode each \ws\ RawData shape (string, Buffer, ArrayBuffer, Buffer[]) explicitly instead of a bare toString(), which would garble ArrayBuffer/Buffer[] frames and silently drop valid RPC messages. - Impact Report: fix two stale comments — the default filter shows all statuses (not differences-only), and the action bar has a connection indicator with auto-reconnect (not a manual reconnect button). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances the Studio "find a regression" journey: a native, per-agent **Impact
Report**, a single coherent connection experience across the sidebar, and a
batch of shared-helper dedupe.
Critical changes
Impact Report — native redesign. Rebuilt into a VS Code-native, per-agent
webview that compares how an agent resolves a corpus across two versions:
working tree) so you can diff a committed version against uncommitted edits.
Compare view with per-side cache state.
explain every value (status terms, resolution tokens, the
A/BmsBase/Compare latency).
One connection experience. The four views (Sandboxes, Corpora, Event Log,
Collisions) drop their per-view Connect/Reconnect buttons for a single
status-bar indicator (e.g. Connected / Reconnecting in 2s…) that
auto-reconnects with backoff. Views show the native loading bar while connecting,
then content or a placeholder.
Tree polish. Trees now render Markdown hover cards and lean on icons for
status (sandbox health, corpus thumbs-up/down feedback) instead of duplicated
text.
Supporting changes
RPC channel, webview nonce, reconnect backoff, agent-name resolution, and a
base class for the tree providers.
hitdistinct from a grammar match, gated on the dispatcher's schema-hash check).
--end-of-optionsso aleading-dash ref can't be parsed as a git option (defense-in-depth on the
host↔git boundary; the webview itself never shells out to git).
Testing
Studio suite green (~180 tests); typecheck and Prettier clean.
19 commits · 60 files.