Skip to content

H-6763: Migrate voice interviews to GPT-Realtime-2 - #9399

Draft
kostandinang wants to merge 3 commits into
kostandin/h-6763-voice-interview-uxfrom
kostandin/h-6763-gpt-realtime-2
Draft

H-6763: Migrate voice interviews to GPT-Realtime-2#9399
kostandinang wants to merge 3 commits into
kostandin/h-6763-voice-interview-uxfrom
kostandin/h-6763-gpt-realtime-2

Conversation

@kostandinang

@kostandinang kostandinang commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

🌟 What is the purpose of this PR?

Migrate the H-6763 voice interview from a transcription-plus-TTS chain to gpt-realtime-2 as an ephemeral duplex media shell, while keeping Brunch the sole durable authority for interview state and business logic. Present canonical Brunch output through a warm, professionally neutral expert-interviewer voice.

🔗 Related links

🚫 Blocked by

🔍 What does this change?

  • Configures a server-owned gpt-realtime-2 WebRTC session with audio input/output, low-effort reasoning, low-eagerness semantic VAD, automatic interruption, and one required continue_interview function.
  • Adds a focused Realtime–Brunch bridge that validates and serializes tool calls, derives stable submission identity, preserves pending brunch_ask correlation, and rejects duplicate, overlapping, or stale work.
  • Lets the first complete spoken description start Brunch when no question is pending, while later turns remain correlated with the active question.
  • Emits brunch_ask as a dynamic AI SDK tool so Petrinaut recognizes and renders each returned question.
  • Keeps the microphone active during Brunch processing and assistant playback, receives remote audio over WebRTC, and exposes orthogonal connection/input/output state to the existing minimal voice UI.
  • Clears provisional transcript text after failed turns and surfaces specific correlation, response, and submission diagnostics.
  • Gives the Realtime session and canonical response requests a consistent expert-interviewer delivery policy without allowing OpenAI to invent, alter, or answer interview questions.
  • Returns canonical Brunch segments through the function result and requests tool-free audio without treating generated speech as the visible source of truth.
  • Removes the separate Speech API route, MP3 playback controller, speech queue, and manual done/interrupt controls.
  • Updates the voice ADR, Petrinaut website setup, and Petrinaut user guide. Transient implementation plans and design artifacts are excluded from the final diff.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

  • Voice remains a non-production preview and is still disabled by server policy in production.
  • The live OpenAI preflight used synthetic browser media in the orb; a physical microphone and real acoustic environment still need preview-deployment validation.

🐾 Next steps

  • Exercise microphone quality, thinking pauses, expert-interviewer delivery, and barge-in with a physical device in the preview deployment.

🛡 What tests cover this?

  • Focused website tests cover Realtime session policy, initial interview startup, pending-question correlation, canonical response delivery, transcript cleanup, diagnostics, reconnect, and cleanup.
  • Brunch transport tests cover dynamic brunch_ask tool streaming and return-path handling.
  • Integration coverage exercises remote media, semantic turn events, automatic interruption, malformed events, failures, and continuous capture.
  • TypeScript checks, ESLint, formatting, architecture docs validation, and the production build pass.
  • A live gpt-realtime-2 WebRTC preflight confirmed remote RTP audio, function calling, semantic VAD, and automatic interruption with the configured key.

❓ How to test this?

  1. Configure a preview or local environment with PETRINAUT_OPENAI_VOICE_ENABLED=true, OPENAI_VOICE_API_KEY, and a working Brunch endpoint.
  2. Start an empty interview, speak a process description, and confirm Brunch starts and returns the first recognized question without an unknown-tool error.
  3. Answer the pending question and confirm the answer reaches that brunch_ask call exactly once.
  4. Confirm the canonical Brunch response remains visible while Realtime speaks it with the expert-interviewer delivery style.
  5. Speak during interviewer audio and confirm playback stops automatically while the microphone remains active.
  6. Exercise failed submission, pause, reconnect, end, and text fallback; confirm provisional text clears after failure and old media/events do not survive reconnect or end.

📹 Demo

No screenshot replacement is required; the affected documentation contains no screenshots of this surface.

@ampagent @kostandinang
Migrate voice interviews to GPT-Realtime-2
e3c137d 
Replace the transcription-only and separate Speech API flow with a server-owned gpt-realtime-2 WebRTC session for duplex media, semantic turn detection, canonical speech, and automatic interruption.

Keep Brunch authoritative through a narrow Realtime bridge that validates and serializes tool calls, preserves pending ask correlation, and rejects stale or duplicate work. Remove the obsolete MP3 playback queue and speech endpoint, expose orthogonal voice state in the UI, and update the voice architecture documentation.

Amp-Thread-ID: https://ampcode.com/threads/T-01a04727-ad92-741e-81b2-4fd81eb53250
Co-authored-by: Kostandin Angjellari <ka@hash.ai>
@vercel

vercel Bot commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview Aug 28, 2026 6:14pm
petrinaut Ready Ready Preview Aug 28, 2026 6:14pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign-tokens Ignored Ignored Preview Aug 28, 2026 6:14pm
petrinaut-docs Ignored Ignored Preview Aug 28, 2026 6:14pm

Request Review

@CLAassistant

CLAassistant commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ kostandinang
❌ ampagent
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/apps labels Aug 28, 2026
@kostandinang Graphite App

kostandinang commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@kostandinang kostandinang self-assigned this Aug 28, 2026
This was referenced Aug 28, 2026
Open
Open
Open
Open
Open
@kostandinang @cursoragent
Improve realtime voice interview startup and delivery
cadbac8 
Ensure initial spoken descriptions reach Brunch, dynamic questions render correctly, and canonical responses use the expert-interviewer voice.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the area/tests New or updated tests label Aug 28, 2026
@kostandinang @cursoragent
Remove transient voice planning documents
37d9a99 
Keep implementation-only planning artifacts out of the product pull request.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@kostandinang kostandinang

Labels

area/apps area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > frontend Owned by the @frontend team

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kostandinang @CLAassistant @ampagent