fix(xai): align realtime Voice Agent defaults and API coverage - #2255
fix(xai): align realtime Voice Agent defaults and API coverage#2255rosetta-livekit-bot[bot] wants to merge 3 commits into
Conversation
🦋 Changeset detectedLatest commit: 3e5bca2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } catch (error) { | ||
| delete this.responseCreatedFutures[eventId]; | ||
| if (!forceMessageSent) this.dropPendingSayTag(eventId); | ||
| if (doneFut.done) return await doneFut.await; | ||
| throw error; |
There was a problem hiding this comment.
🔴 Scripted speech stops working after the voice connection reconnects
A scripted-speech request that is still waiting when the connection drops leaves its tracking tag in the pending queue forever (the tag is only removed when the message was never sent, at plugins/xai/src/realtime/realtime_model.ts:216), so every later scripted-speech request is matched to the wrong reply and eventually fails.
Impact: After a reconnect, say() on xAI realtime times out (~10s) and the reply that should have been spoken is cancelled instead, so the agent stays silent.
Pending say tag queue desynchronizes because reconnect clears futures but not the FIFO tag list
On reconnect the base session rejects all pending response futures with Session reconnected and clears both responseCreatedFutures and discardedEventIds (plugins/openai/src/realtime/realtime_model.ts:1103-1110). In the xAI subclass, the rejected say() lands in the catch block: since forceMessageSent is true, dropPendingSayTag(eventId) is skipped and no stale-cleanup timer is scheduled (scheduleStaleSayCleanup is only invoked from discardSay and the timeout path, plugins/xai/src/realtime/realtime_model.ts:202-210,255-263). The dead id therefore stays in pendingSayEventIds indefinitely.
The next say() pushes its own id behind the stale one. When the server sends response.created, handleResponseCreated shifts the stale id and stamps it as client_event_id (plugins/xai/src/realtime/realtime_model.ts:322-328); the base handler finds no matching future, so the new say() never resolves and rejects after 10s, at which point its id is added to discardedEventIds and the following response is cancelled.
A fix would clear pendingSayEventIds (and staleSayTimers) whenever the session reconnects, and drop the tag on any terminal failure of a sent say, not just aborts before sending.
Prompt for agents
In plugins/xai/src/realtime/realtime_model.ts, the FIFO correlation list pendingSayEventIds can retain dead entries. When the underlying OpenAI base session reconnects it rejects every entry in responseCreatedFutures with 'Session reconnected' and clears responseCreatedFutures/discardedEventIds, but nothing clears the xAI-side pendingSayEventIds or staleSayTimers. Because say()'s catch block only drops the pending tag when the force_message was never sent (forceMessageSent === false), a say that was in flight at reconnect time leaves its id in pendingSayEventIds permanently. handleResponseCreated then shifts that dead id onto the next response.created, so subsequent say() calls never resolve (they time out after 10s) and the response after that gets cancelled as 'discarded'. Consider hooking the session-reconnected path (or overriding whatever runs on reconnect) to reset pendingSayEventIds/staleSayTimers, and make say() drop or schedule cleanup of its tag on any terminal rejection, not only on pre-send aborts.
Was this helpful? React with 👍 or 👎 to provide feedback.
| say( | ||
| _text: string | ReadableStream<string>, | ||
| _options: { signal?: AbortSignal } = {}, | ||
| ): Promise<GenerationCreatedEvent> { | ||
| throw new Error(`${this.constructor.name} does not implement say(); use a TTS model instead`); | ||
| } |
There was a problem hiding this comment.
🟡 New public methods and protocol types are added without documentation comments
The newly added scripted-speech method on the realtime session base class (say() at agents/src/llm/realtime.ts:150) has no documentation comment, which the repository's contribution rules require for every new method, interface or class.
Impact: The generated API documentation is missing entries for newly exposed public API.
Rule source and other affected declarations
CONTRIBUTING.md: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."
Undocumented new declarations in this PR:
RealtimeSession.say()—agents/src/llm/realtime.ts:150-155ForceMessageItemCreate—plugins/openai/src/realtime/api_proto.ts:395-399ForceMessageCreateEvent—plugins/openai/src/realtime/api_proto.ts:427-430
(The new supportsSay capability flag is correctly documented.)
| say( | |
| _text: string | ReadableStream<string>, | |
| _options: { signal?: AbortSignal } = {}, | |
| ): Promise<GenerationCreatedEvent> { | |
| throw new Error(`${this.constructor.name} does not implement say(); use a TTS model instead`); | |
| } | |
| /** | |
| * Speak the given scripted text directly through the realtime session. | |
| * | |
| * Only supported by providers that advertise {@link RealtimeCapabilities.supportsSay}; | |
| * the default implementation throws. | |
| * | |
| * @param _text - The text (or text stream) to speak. | |
| * @param _options - Optional abort signal used to cancel the request. | |
| */ | |
| say( | |
| _text: string | ReadableStream<string>, | |
| _options: { signal?: AbortSignal } = {}, | |
| ): Promise<GenerationCreatedEvent> { | |
| throw new Error(`${this.constructor.name} does not implement say(); use a TTS model instead`); | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| private discardSay(eventId: string): void { | ||
| delete this.responseCreatedFutures[eventId]; | ||
| if (!this.discardedEventIds.has(eventId)) { | ||
| this.sendEvent({ type: 'response.cancel' }); | ||
| this.discardedEventIds.add(eventId); | ||
| this.scheduleStaleSayCleanup(eventId); | ||
| } | ||
| this.ensurePendingSayTag(eventId); | ||
| } |
There was a problem hiding this comment.
🟡 Cancelled scripted speech can silently suppress the agent's reply to the user
When a scripted speech request is cancelled after it was sent, its tracking tag is deliberately kept in the pending queue (ensurePendingSayTag at plugins/xai/src/realtime/realtime_model.ts:262), so whichever reply the server produces next — including a genuine reply to the user — is thrown away.
Impact: For up to 10 seconds after a cancelled scripted line, the agent can silently drop its answer to what the user just said.
Mechanism: discarded tag is applied to an arbitrary next response
discardSay adds the event id to discardedEventIds and re-inserts it into pendingSayEventIds (plugins/xai/src/realtime/realtime_model.ts:255-263). handleResponseCreated stamps the head of pendingSayEventIds onto any incoming response.created that lacks a client_event_id (plugins/xai/src/realtime/realtime_model.ts:322-328). With server VAD (create_response: true, the xAI default at plugins/xai/src/realtime/realtime_model.ts:21-28), the server autonomously creates a response as soon as the user finishes speaking. If such a response is the first one to arrive after the cancellation, the base handler sees a discarded id and cancels/discards it (plugins/openai/src/realtime/realtime_model.ts:1462-1472), so the user's turn goes unanswered. The tag is only released by the 10s cleanup timer (plugins/xai/src/realtime/realtime_model.ts:265-277).
A safer correlation would only discard responses whose creation timestamp/order can actually be attributed to the cancelled force_message (e.g. discard only until the very next response, and drop the tag as soon as any response has been observed).
Was this helpful? React with 👍 or 👎 to provide feedback.
e89bf78 to
f1d4957
Compare
| const recoverable = !isFatalError(event.error); | ||
| const error = new APIError(event.error.message, { | ||
| body: event.error, | ||
| retryable: recoverable, | ||
| }); | ||
| if (!recoverable) throw error; | ||
| this.emitError({ error, recoverable: true }); |
There was a problem hiding this comment.
🔴 A billing or credential error from the voice provider now crashes the whole agent process
A fatal server error is re-thrown from inside the WebSocket message callback (throw error at plugins/openai/src/realtime/realtime_model.ts:2054) instead of being reported through the session's error channel, so the failure escapes as an uncaught exception and takes the process down.
Impact: Errors such as an exhausted quota or an invalid API key kill the running agent worker instead of surfacing a normal session error to the application.
Mechanism: throw escapes the ws `onmessage` handler with no catch
handleError is invoked synchronously from the wsConn.onmessage handler registered in runWs (plugins/openai/src/realtime/realtime_model.ts:1254-1337). That handler is called by the ws EventEmitter while draining socket data; nothing wraps it in a try/catch, and runWs's own try/finally only covers the awaited task promises, not the synchronous event callback. Therefore throw error propagates to Node as an uncaughtException.
Additionally, because the throw happens before this.emitError(...), the realtime_model_error event is never emitted for fatal codes (insufficient_quota, invalid_api_key, account_deactivated, billing_hard_limit_reached — see plugins/openai/src/realtime/realtime_model.ts:129-138), so AgentActivity.onError never sees it and the reconnect/retry logic in #mainTask is bypassed as well.
A safer shape is to emit a non-recoverable error (emitError({ error, recoverable: false })) and let the main task tear the session down, rather than throwing from the socket callback.
| const recoverable = !isFatalError(event.error); | |
| const error = new APIError(event.error.message, { | |
| body: event.error, | |
| retryable: recoverable, | |
| }); | |
| if (!recoverable) throw error; | |
| this.emitError({ error, recoverable: true }); | |
| const recoverable = !isFatalError(event.error); | |
| const error = new APIError(event.error.message, { | |
| body: event.error, | |
| retryable: recoverable, | |
| }); | |
| this.emitError({ error, recoverable }); |
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
grok-voice-latestwithgrok-transcribesay()support using xAIforce_message, including FIFO response correlation and cancellation cleanupSource diff coverage
Source diff coverage
livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/__init__.py: not applicable.plugins/xai/src/realtime/index.tsalready exports the target-nativeRealtimeSessionimplementation.livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.py: adapted toplugins/xai/src/realtime/realtime_model.ts, with the missing realtimesupportsSay/say()voice-pipeline infrastructure added inagents/src/llm/realtime.tsandagents/src/voice/agent_activity.ts, and reusable protocol/session seams added inplugins/openai/src/realtime/api_proto.tsandplugins/openai/src/realtime/realtime_model.ts. Python futures/tasks are represented with JS promises, abort signals, and stream cancellation.livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/types.py: ported to the colocatedGrokRealtimeModelstype inplugins/xai/src/realtime/realtime_model.ts.tests/test_realtime/test_realtime.py: adapted toplugins/xai/src/realtime/realtime_model.test.ts. The target has no shared credential-gated realtime provider/WAV harness; its existing xAI chat-context test already covers history deletion, and the sourcesay()smoke is covered at the wire/generation boundary.tests/test_realtime/test_xai_realtime_model.py: ported toplugins/xai/src/realtime/realtime_model.test.ts, including defaults, session updates, live captions,say()cancellation/close handling, and FIFO correlation.Test plan
env -u LIVEKIT_URL -u LIVEKIT_API_KEY -u LIVEKIT_API_SECRET pnpm vitest run agents(1065 passed, 2 skipped)env -u OPENAI_API_KEY pnpm vitest run plugins/openai(56 passed, 7 skipped)pnpm vitest run plugins/xai(29 passed, 1 skipped)pnpm buildpnpm lint(passes with existing warnings)pnpm format:checkValidation notes
cue-clidispatched the xAI realtime voice agent and reachedAgentActivity.realtime_say; the configured xAI endpoint rejected the WebSocket handshake with HTTP 400 before framework assistant events could be emitted.export * asdeclarations and a missingplugins/bey/api-extractor.json; the touched packages build and emit declarations successfully.Upstream: livekit/agents#6755
Ported from livekit/agents#6755
Original PR description
Summary
grok-voice-latest(resolves togrok-voice-think-fast-2.0) and default input transcription togrok-transcribeforce_messagesupport viaRealtimeSession.say()/supports_say, emit live caption updates fromconversation.item.input_audio_transcription.updated, and forwardreasoning/ transcription / speed optionslivekit-xai-issues.mdTest plan
uv run pytest tests/test_realtime/test_xai_realtime_model.py -q --allow-uncategorized(20 passed)XAI_API_KEY=… uv run pytest tests/test_realtime/test_realtime.py -k xai -q --allow-uncategorized(16 passed)grok-voice-latest,session.update(reasoning / idle_timeout / grok-transcribe), cancel, force_message, deleteMade with Cursor