H-6763: Add realtime voice interviewing experiments - #9297
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
github-actions
Bot
added
area/infra
|
|
![semgrep-code-hashintel[bot]](https://avatars.githubusercontent.com/u/22116112?s=60&v=4){kind=small}
| ) => { | ||
| // Never add request bodies, API keys, or upstream response bodies here. | ||
| // oxlint-disable-next-line no-console | ||
| console.error(`[OpenAI Realtime experiment] ${reason}`, context); |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
Detected string concatenation with a non-literal variable in a util.format / console.log function. If an attacker injects a format specifier in the string, it will forge the log message. Try to use constant values for the format string.
Dataflow graph
flowchart LR
classDef invis fill:white, stroke: none
classDef default fill:#e7f5ff, color:#1c7fd6, stroke: none
subgraph File0["<b>apps/petrinaut-website/api/voice-experiment/openai-realtime-session.ts</b>"]
direction LR
%% Source
subgraph Source
direction LR
v0["<a href=https://github.com/hashintel/hash/blob/3488a1fe5689cbb8140641d72794614556b0d1f7/apps/petrinaut-website/api/voice-experiment/openai-realtime-session.ts#L115 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 115] `[OpenAI Realtime experiment] ${reason</a>"]
end
%% Intermediate
%% Sink
subgraph Sink
direction LR
v1["<a href=https://github.com/hashintel/hash/blob/3488a1fe5689cbb8140641d72794614556b0d1f7/apps/petrinaut-website/api/voice-experiment/openai-realtime-session.ts#L115 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 115] `[OpenAI Realtime experiment] ${reason</a>"]
end
end
%% Class Assignment
Source:::invis
Sink:::invis
File0:::invis
%% Connections
Source --> Sink
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by unsafe-formatstring.
You can view more details about this finding in the Semgrep AppSec Platform.
| const OPENAI_CLIENT_SECRETS_URL = | ||
| "https://api.openai.com/v1/realtime/client_secrets"; |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
A hardcoded secret is identified. Store it properly in an environment variable.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by node_secret.
You can view more details about this finding in the Semgrep AppSec Platform.
| const originalVercelEnvironment = process.env.VERCEL_ENV; | ||
|
|
||
| beforeEach(() => { | ||
| process.env.OPENAI_API_KEY = "primary-secret-that-must-stay-server-side"; |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
A hardcoded API Key is identified. Store it properly in an environment variable.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by node_api_key.
You can view more details about this finding in the Semgrep AppSec Platform.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ) => { | ||
| // Never add request bodies, API keys, or upstream response bodies here. | ||
| // oxlint-disable-next-line no-console | ||
| console.error(`[OpenAI Realtime experiment] ${reason}`, context); |
github-actions
Bot
added
area/deps
| ) => { | ||
| // Never add request bodies, API keys, or upstream response bodies here. | ||
| // oxlint-disable-next-line no-console | ||
| console.error(`[ElevenLabs voice experiment] ${reason}`, context); |
2bbb360 to
27fa5c5
Compare
| const SESSION_ENDPOINT = "/api/voice-experiment/openai-realtime-session"; | ||
| const REALTIME_CALLS_ENDPOINT = "https://api.openai.com/v1/realtime/calls"; | ||
| const CONNECTION_TIMEOUT_MS = 20_000; | ||
| const WAIT_FOR_USER_TOOL_NAME = "wait_for_user"; |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
A hardcoded username in plain text is identified. Store it properly in an environment variable.
To resolve this comment:
🔧 No guidance has been designated for this issue. Fix according to your organization's approved methods.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by node_username.
You can view more details about this finding in the Semgrep AppSec Platform.
| operation: Operation, | ||
| productName: string = PRODUCT_NAME, | ||
| ): string { | ||
| if (!IDENTIFIER.test(operation)) { |
There was a problem hiding this comment.
Ensure that the regex used to compare with user supplied input is safe from regular expression denial of service.
🍰 Removed in commit eaecb5e 🍰
64c8ee7 to
eaecb5e
Compare
| const SESSION_ENDPOINT = "/api/voice-experiment/openai-realtime-session"; | ||
| const REALTIME_CALLS_ENDPOINT = "https://api.openai.com/v1/realtime/calls"; | ||
| const CONNECTION_TIMEOUT_MS = 20_000; | ||
| const WAIT_FOR_USER_TOOL_NAME = "wait_for_user"; |
Permit the regular Petrinaut development origins so OpenAI voice sessions can reach the real Brunch elicitor through the local proxy. Co-authored-by: Cursor <cursoragent@cursor.com>
Create and persist a mock Petrinaut net from completed voice interviews so provider and elicitor combinations can be validated end to end. Co-authored-by: Cursor <cursoragent@cursor.com>
Make the URL parameter describe the component that projects interview evidence into a net, independently of the elicitor. Co-authored-by: Cursor <cursoragent@cursor.com>
Expose settled elicitation results as a revisioned live draft so both voice integrations can validate projection before the interview finishes. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
PR SummaryMedium Risk Overview OpenAI Realtime gets server-minted ephemeral credentials and fixed server-owned session presets (mock capture tools vs Brunch speech-renderer). The browser adapter admits only finalized expert transcripts into Brunch when ElevenLabs + Brunch adds a local Speech Engine server in The voice experiment panel (launcher, transcript, tool diagnostics, logs, live mock projection) wires into local storage so projected nets persist transcript, revision, and warnings, with guards against stale revisions and manual edits. Provider API routes enforce same-origin / experiment headers, empty bodies (no browser-controlled config), rate limits, and generic error responses without leaking keys. Reviewed by Cursor Bugbot for commit 2012f05. Bugbot is set up for automated code reviews on this repo. Configure here. |
Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1f8fd3a. Configure here.
Co-authored-by: Cursor <cursoragent@cursor.com>
3 participants
🌟 What is the purpose of this PR?
This draft prototypes realtime voice interviewing of domain experts in Petrinaut and separates the voice provider, elicitor, and mock projector so their integration boundaries can be evaluated independently.
It supports three interview configurations:
Any supported configuration can add
projector=mockto exercise incremental and final handoff into a persisted Petrinaut net. The intent is to compare providers against the same elicitor without moving authoritative conversation state, structured questions, or elicitation policy out of Brunch.🔗 Related links
libs/@hashintel/brunch-agent/docs/planning/process-model-elicitation/spikes/h-6763-realtime-audio-prototype-2026-08-24.mdapps/petrinaut-website/src/main/app/local-storage-demo/voice-experiment/README.md☁️ Preview deployment
For the Vercel preview, set
OPENAI_API_KEY,ELEVENLABS_API_KEY, andELEVENLABS_SPEECH_ENGINE_IDfor the Petrinaut project’s Preview environment, then redeploy. Keep provider keys server-only; do not expose them asVITE_*variables.OpenAI + mock works with the OpenAI key alone. Brunch modes additionally need a deployed
/api/voice-experiment/brunch-chatrelay to Brunch, and ElevenLabs needs its public Speech Enginewss://…/wsbridge.🔍 What does this change?
voiceProvider,elicitor, and optionalprojectorURL parameters. LegacyvoiceExperimentlinks remain supported.brunch_askcorrelation flow through the existing AI SDK transport; OpenAI renders the resulting text as out-of-band audio.BrunchVoiceBridgeinto the private AI SDK transport workspace package so both voice integrations share pending-ask and applied-sweep behavior.brunch_sweepresults update one revisioned live net; finishing forces a final projection and falls back to a labelled placeholder when evidence is incomplete.🏗️ Agent notes
The provider and elicitor are fixed for the lifetime of the page. Changing either requires a reload and a new conversation so provider history, pending asks, queued audio, and microphone state cannot leak between comparisons.
Supported interview selections are:
?voiceProvider=openai&elicitor=mock?voiceProvider=openai&elicitor=brunch?voiceProvider=elevenlabs&elicitor=brunchAdd
&projector=mockto any selection to exercise live and final mock-net projection.elevenlabs&elicitor=mockis rejected because each Speech Engine conversation is bound to its server-side callback and the current resource is wired to Brunch.In both real-elicitor paths, Brunch owns authoritative history, elicitation policy,
brunch_ask, tool activity, and evidence admission. Provider partial transcripts are display-only; only finalized expert input enters Brunch.OpenAI Realtime is used as a speech renderer for Brunch output, but model-generated speech is not guaranteed to be verbatim. The UI therefore preserves the exact Brunch response as authoritative and suppresses generated paraphrases from the visible transcript.
The mock projector validates the integration contract; it is not the production Brunch read-time projector. Structured experiment captures produce a state-step-flow graph. An applied Brunch sweep emits readiness and can produce the best available mock draft, while refused sweeps do not trigger projection.
Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
The shared bridge export is added to a private workspace package. The Petrinaut library touch only lets the experimental voice dock coexist with Ask AI and does not add a published API.
📜 Does this require a change to the docs?
The changes in this PR:
Current provider configuration, URL selection, projector behavior, local run instructions, security boundaries, and known caveats are documented in
apps/petrinaut-website/src/main/app/local-storage-demo/voice-experiment/README.md. The linked spike remains the original prototype plan rather than the final comparison record.🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
The website already depended on the private AI SDK transport package; this adds a build/export entry without changing workspace topology or
turbo.json.wss://…/wsendpoint for the local server.🐾 Next steps
🛡 What tests cover this?
brunch_askcontinuation, streamed responses, cancellation, revised-turn serialization, structured capture diagnostics, and applied-versus-refused sweep handling.❓ How to test this?
Follow
apps/petrinaut-website/src/main/app/local-storage-demo/voice-experiment/README.mdto configure local provider values.Test native OpenAI mock mode at
/?voiceProvider=openai&elicitor=mock.Start the real Brunch server, then test OpenAI against Brunch at
/?voiceProvider=openai&elicitor=brunch. Confirm finalized expert speech reachesvoice:<conversation-id>, Brunch's response is displayed and spoken, and the microphone reopens only after playback drains.For ElevenLabs, also start the
voice:devSpeech Engine server, expose/wsthrough a public TLS tunnel, configure the resource URL, and open/?voiceProvider=elevenlabs&elicitor=brunch.In both Brunch modes, complete at least one
brunch_asksuspend/resume cycle and confirm tool activity is visible without duplicate turns.Repeat the supported URLs with
&projector=mock. Confirm coherent captures or an applied sweep create one live draft, later revisions update that draft, manual edits are not overwritten, and Finish and create net forces a final result.Run the focused checks:
📹 Demo
Watch the voice interview experiment demo on Loom