H-6763: Add OpenAI Realtime voice input to Brunch

[kostandinang]

🌟 What is the purpose of this PR?

Add disabled-by-default OpenAI Realtime voice input to Brunch inside the Petrinaut website. OpenAI transcribes microphone audio; each completed transcript is submitted through Petrinaut's generic composer API and the existing AI SDK/Brunch transport.

Brunch remains authoritative for tools, pending asks, evidence, completion, and projection. OpenAI is used only for transcription in this PR.

🔗 Related links

• Linear: H-6763 (internal)
• Depends on #9355
• Stack: #9346 → #9355 → H-6763: Add OpenAI Realtime voice input to Brunch #9356 → #9357
• #9297 is historical evidence only and is unchanged.

🚫 Blocked by

• #9355, transitively #9345 and #9346

🔍 What does this change?

• Adds one app-owned, same-origin WebRTC initialization boundary.
  • The browser sends only its SDP offer.
  • The server atomically adds the OpenAI key and fixed transcription policy before calling /v1/realtime/calls.
  • Provider credentials, model, language, vocabulary, and VAD policy never reach the browser.
• Uses transcription-only gpt-live-transcribe with English process-modeling vocabulary and semantic VAD at low eagerness.
  • No Realtime response generation is enabled.
  • Voice is unavailable unless the server flag and dedicated key are both present.
  • Public production fails closed; this remains a deployment-protected preview feature.
• Implements the browser WebRTC lifecycle: microphone permission, media tracks, peer connection, data channel, connection timeout, cleanup, reconnect epochs, and stale-callback rejection.
• Admits only completed transcript events.
  • Partial transcripts are visible but never submitted.
  • Completed turns are deduplicated by connection epoch, OpenAI item ID, and content index.
  • Stable message IDs preserve one semantic submission across transport retries.
• Routes finalized text through H-6763: Add a generic Petrinaut composer submission API #9355 and the existing Brunch transport.
  • A pending brunch_ask remains correlated and receives the answer through its tool call.
  • Explicit corrections are separate normal messages and cannot accidentally answer a pending ask.
• Adds a half-duplex microphone controller and accessible UI for start, end, reconnect, provisional text, correction, and status.
  • The microphone stays closed while connecting, finalizing, delivering, or waiting for Brunch.
  • Keyboard chat remains available on every failure path.
• Installs voice only for the real Brunch endpoint; the generic fallback chat remains voice-free.
• Updates preview configuration, deployment limits, and user/developer documentation without adding a provider abstraction or SDK dependency.

Stack record: direct base 3152049cabe28aed98c4e80560366f16fcaae0e4; PR head 7c042206e919b80ef27ce46649bfe5202005b27c.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

No existing documentation screenshot covers the new voice control.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Voice is disabled by default and intentionally unavailable in public production.
• Real provider/browser validation still requires a dedicated OpenAI project, microphone permission, and a real remote Brunch endpoint.
• The inherited Brunch docs-index check remains 6/7 because FE-1506: Show captures and completion while a panel elicitation runs #9346 references ADR-0008 from the parallel prerequisite stack.

🐾 Next steps

• H-6763: Speak finalized Brunch responses with OpenAI #9357 speaks finalized canonical Brunch text through OpenAI's dedicated Speech API.
• Voice PR4 will add delivery/reload recovery, stale-ask handling, authentication, distributed quotas, telemetry, retention cleanup, a kill switch, and real completion/projection evidence after the required platform contracts land.

🛡 What tests cover this?

• Server tests cover fail-closed configuration, production blocking, same-origin/size/content validation, server-owned policy, sanitized upstream failures, browser aborts, and timeouts.
• WebRTC tests cover complete lifecycle cleanup, microphone permission failures, reconnect epochs, strict OpenAI event parsing, and stale events.
• Turn-controller tests cover partial/final separation, stable identity, duplicate/out-of-order events, half-duplex races, stop/reconnect, pending-ask routing, and corrections.
• Component/host tests cover accessible status and installation only on the real Brunch path.

Validation run

yarn workspace @hashintel/petrinaut test:unit --run
yarn workspace @hashintel/petrinaut lint:eslint
yarn workspace @hashintel/petrinaut lint:tsc
yarn workspace @hashintel/petrinaut build
yarn workspace @apps/petrinaut-website test:unit
yarn workspace @apps/petrinaut-website lint:eslint
yarn workspace @apps/petrinaut-website lint:tsc
yarn workspace @apps/petrinaut-website build
yarn workspace @hashintel/brunch-agent-transport-aisdk test:unit
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:eslint
yarn workspace @hashintel/brunch-agent-transport-aisdk lint:tsc
yarn workspace @hashintel/brunch-agent-transport-aisdk build
yarn workspace @apps/brunch-agent test:unit
yarn workspace @apps/brunch-agent lint:eslint
yarn workspace @apps/brunch-agent lint:tsc
yarn workspace @apps/brunch-agent build
yarn workspace @local/petrinaut-arch-docs lint:arch-docs
yarn oxfmt --check apps/petrinaut-website apps/brunch-agent libs/@hashintel/petrinaut libs/@hashintel/brunch-agent

Results: 214 Petrinaut tests, 49 website tests, 12 transport tests, and 35 Brunch tests passed. Lint, typecheck, build, formatting, and architecture-doc checks passed. Transport emitted 4 inherited no-await-in-loop warnings and Brunch emitted 12 inherited warnings.

The direct-base no-ElevenLabs audit and git diff --check passed. No package manifest or lockfile changed.

Credential-dependent checks not run: live WebRTC negotiation, representative-audio semantic VAD, one finalized item reaching remote Brunch/pending ask, live reconnect rejection, desktop Chrome/Edge/Safari accessibility, and deployed production unavailability.

❓ How to test this?

1. Run yarn workspace @apps/petrinaut-website test:unit plus its lint, typecheck, and build commands above.
2. In a protected preview with PETRINAUT_OPENAI_VOICE_ENABLED=true, a dedicated server-side OPENAI_VOICE_API_KEY, and a real Brunch endpoint, start voice input and speak one answer.
3. Confirm provisional text is labelled “not sent,” only the completed transcript reaches the current Brunch turn, and the microphone remains off until Brunch is ready again.
4. End/reconnect voice and confirm late events do not submit; disable the feature and confirm keyboard chat still works.

📹 Demo

Not included: a meaningful demo requires provider credentials, microphone permission, and the remote Brunch environment described above.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hash	Ready	Preview	Aug 27, 2026 12:43pm
petrinaut	Ready	Preview	Aug 27, 2026 12:43pm
petrinaut-docs	Ready	Preview	Aug 27, 2026 12:43pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
hashdotdesign-tokens	Ignored	Preview	Aug 27, 2026 12:43pm

[kostandinang]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• H-6763: Add the voice interview experience #9380
• H-6763: Improve voice preview reliability diagnostics #9378
• H-6763: Speak finalized Brunch responses with OpenAI #9357
• H-6763: Add OpenAI Realtime voice input to Brunch #9356 👈 (View in Graphite)
• H-6763: Add a generic Petrinaut composer submission API #9355
• FE-1507: Move the repertoire into core behind a guarded prompts subpath #9349 : 1 other dependent PR (#9381 )
• FE-1431: Build the plugin authoring surface (cycle two) #9348
• FE-1505: Time each interviewer turn by purpose #9347
• FE-1506: Show captures and completion while a panel elicitation runs #9346
• FE-1504: Run the process-model elicitor from the Petrinaut panel on one owned document #9345
• FE-1431: Build the plugin authoring surface #9337
• FE-1406: Audit the harness-teaching lineage and propose the fixed-key plugin contract (ADR-0007) #9327
• FE-1497: Build the controller read path and the SDCPN plugin package #9325
• FE-1404: Run condition 3 as the walking skeleton #9322
• FE-1403: Assemble and test the CPS interview guidance #9321
• FE-1402: Define and rehearse the elicitation completion contract #9320
• FE-1407: Catalogue elicitor failures that published measures miss #9319
• FE-1480: Project and realize a CPS model as a Petrinaut SDCPN #9313
• FE-1495: Prevent Brunch TypeScript contracts from drifting #9312
• FE-1494: Keep Brunch strategy and control records coherent #9311
• FE-1437: Move brunch-agent into hashintel/hash with its history #9274
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Introduces WebRTC, microphone access, and a server proxy that holds an OpenAI key; mitigated by production fail-closed gating, origin checks, and transcription-only policy, but preview misuse or quota exposure remains possible without auth.

Overview
Adds a disabled-by-default OpenAI Realtime voice-input preview to the Petrinaut website when VITE_BRUNCH_CHAT_ENDPOINT points at Brunch; the local /api/chat fallback stays voice-free.

Server: New /api/voice/config and /api/voice/realtime-call routes proxy WebRTC setup: the browser posts only an SDP offer; the server attaches a dedicated OPENAI_VOICE_API_KEY and a fixed transcription-only gpt-live-transcribe policy, then calls OpenAI. Voice is off unless PETRINAUT_OPENAI_VOICE_ENABLED=true and a key are set, and it fails closed in production. Requests are same-origin validated with sanitized errors.

Client: WebRTC session lifecycle, a half-duplex VoiceTurnController (mic off while Brunch is busy; only completed transcripts submit via Petrinaut submitText), and an accessible composer control with corrections. Brunch preview config adds stable per-net conversationId; brunch_ask gains fromComposerText so voice can answer pending asks.

Dev/Vercel wiring, env/docs, and ADR note on using default server VAD during Realtime init are updated.

^{Reviewed by Cursor Bugbot for commit 824f0a9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 20c3818. Configure here.}