[azure-ai-projects] Add Voice Agents realtime client, samples, and tests aligned with .beta structure - #48965
Conversation
…ta structure feature/azure-ai-projects/vnext already has the generated Voice Agents surface (.beta.agent_endpoint_conversations, .beta.agent_telephony, .beta.agents telephony binding/call/generate methods, .beta.voice_agent_web_socket) via PR #48950. This change adds the missing hand-written realtime WebSocket client and the surrounding samples/tests/docs to make Voice Agents a complete, usable feature, all under the .beta naming convention: - azure/ai/projects/_realtime.py and aio/_realtime.py: hand-written realtime WebSocket client (Realtime/AsyncRealtime, RealtimeConnection(Manager)), exposed as client.beta.realtime / async_client.beta.realtime via a new property on BetaOperations. Exported from azure.ai.projects.operations / azure.ai.projects.aio.operations (not the top-level package), matching how other .beta-only classes are exported. - 11 samples under samples/agents/voice/: CRUD lifecycle, guided generation, versioning, tools, live text/audio conversations, function-tool calling, and reading back persisted conversations/audio. - 14 test files under tests/agents/ and tests/foundry_features_header/: CRUD, telephony, telephony campaigns, conversations, realtime client (mocked + live), and telephony protocol tests. - Supporting test infra: conftest.py Foundry-Features/Accept sanitizers, test_base.py foundry_voice_model_name, and a NON_OPERATION_BETA_ATTRIBUTES exclusion in the generic .beta header-injection test (realtime is a hand-written WebSocket entry point, not a generated REST operation, so it can't be exercised by that generic mechanism -- it has its own dedicated header test in test_realtime_client.py). - PostEmitter.ps1: regression guard so a future tsp-client update can't silently overwrite _realtime.py's SDK client-identification fix (PR #48848) without failing the emit step. - pyproject.toml: optional ealtime extra (websockets / aiohttp). - CHANGELOG.md, README.md, .env.template, dev_requirements.txt updated. Renames applied while porting (vnext's TypeSpec commit is newer than the source branch this was ported from, and renamed several methods beyond just .beta nesting -- verified against vnext's own generated code and docs/public-methods.md): - agent_endpoint_conversations: dropped the "agent_conversation" infix and renamed *_content methods to download_* (e.g. get_agent_conversation_item -> get_item, get_agent_conversation_item_audio_content -> download_item_audio). - agent_telephony: dropped the redundant "telephony_" infix (e.g. create_telephony_call_job -> create_call_job, begin_validate_telephony_campaign -> begin_validate_campaign). - agents.generate_agent -> agents.generate, and telephony binding/call methods moved from top-level client.agents to client.beta.agents. Validated: black/pylint/mypy/pyright all clean. Full test suite: 219 passed/74 skipped/0 failed outside voice; voice+header suite 732 passed/49 skipped/8 failed (all 8 are brand-new tests with no recording yet, not a regression). Live-tested all 11 samples end-to-end against a real Foundry project with a real gpt-realtime deployment: real HTTP status codes, real multi-turn realtime conversations with byte-accurate audio sizing, real function-tool invocation, and real conversation/audio persistence+readback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
[Pilot] PR Pipeline Failure AnalysisWhat failedPipeline build 6812287 failed on all platforms (macos311, Ubuntu2404_313, ubuntu2404_310, ubuntu2404_310_coverage, Ubuntu2404_314, windows2022_312), each with two failure sources for azure-ai-projects:
Relevant pipeline outputRecommended next steps
Automated fix: Fix found, view and apply fix
|
| model=model, | ||
| instructions="You are a friendly voice assistant. Keep replies short and natural.", | ||
| audio=VoiceAgentAudioConfig( | ||
| output=VoiceAgentAudioOutputConfig(voice="en-US-AvaNeural", voice_type=VoiceType.AZURE_STANDARD), |
There was a problem hiding this comment.
Regarding voice="en-US-AvaNeural" I expected to see a selection from an enum instead of writing a string. Do we defined all voices as a union in TypeSpec? If not, why?
There was a problem hiding this comment.
I checked the generated model: voice is intentionally Optional[str], while voice_type is already an extensible VoiceType enum. I think service added this is also to avoid unnecessary API version changes when voices are added or removed. Since this comes from TypeSpec, we can revisit it at the spec level if needed.
| DefaultAzureCredential() as credential, | ||
| AIProjectClient(endpoint=endpoint, credential=credential, allow_preview=True) as project_client, | ||
| ): | ||
| agent = project_client.beta.agents.generate(GenerateVoiceAgentRequest(kind=AgentKind.VOICE, name=agent_name)) |
There was a problem hiding this comment.
".agents.generate" is a too generic name. We'll see where it lands in our sub-client discussions. Depending on what sub-client it's on, we may or may not need a more descriptive name. If it's ".beta.voice-agent.generate" then the name is probably okay.
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
There was a problem hiding this comment.
Makes sense -- this one's out of my hands in this PR though: .beta.agents.generate is already the shape that landed in the generated code this branch picked up (from #48950). I’ll summarize the fixes related to the TypeSpec.
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
Agreed — kind can only ever be voice here, so it shouldn't need to be passed. Same root cause as the discriminator comment above: TypeSpec currently defines kind as a plain required field instead of a discriminator (or a field with a default), so the generator makes it a required constructor arg. Fixing it needs a TypeSpec change; I'll flag it there rather than hand-patch the generated model here.
| DefaultAzureCredential() as credential, | ||
| AIProjectClient(endpoint=endpoint, credential=credential, allow_preview=True) as project_client, | ||
| ): | ||
| agent = project_client.beta.agents.generate(GenerateVoiceAgentRequest(kind=AgentKind.VOICE, name=agent_name)) |
There was a problem hiding this comment.
Also can we eliminate kind as parameter?
| # session's conversation can be fetched back by id afterward. Reuse the generated | ||
| # definition as-is (instead of reconstructing a new one from a few fields) so audio, | ||
| # greeting, tools, and any other service-selected settings are preserved. | ||
| definition.store = True # type: ignore[attr-defined] |
There was a problem hiding this comment.
Why can't I just pass "store=True" to GenerateVoiceAgentRequest() above and get the agent I want, instead of creating one version without store, then another version with store?
There was a problem hiding this comment.
My bad, generate() has no store field on purpose — it's meant to stay minimal. I'd used generate() + a second create_version() while testing, and it's still in the sample.
Fixed: it now calls create_version() directly with VoiceAgentDefinition(..., store=True) in one step, matching sample_voice_agent_basic.py.
| # definition as-is (instead of reconstructing a new one from a few fields) so audio, | ||
| # greeting, tools, and any other service-selected settings are preserved. | ||
| definition.store = True # type: ignore[attr-defined] | ||
| project_client.agents.create_version( |
There was a problem hiding this comment.
should create_version internally calls generate, and make generate private?
There was a problem hiding this comment.
Good question! I'd keep them separate — generate is for AI-assisted scaffolding, create_version needs to stay generic across all agent kinds. Also I will use create agent api here, not generate and change version. Will update samples
| foundry_features: str = _VOICE_AGENT_FEATURE_HEADER, | ||
| agent_session_id: Optional[str] = None, | ||
| agent_version_override: Optional[str] = None, | ||
| structured_inputs: Optional[str] = None, |
There was a problem hiding this comment.
the name seems the type should be an array. But in the docstr, it should be JSON.
There was a problem hiding this comment.
Good catch, real bug — it was typed Optional[str], silently requiring callers to pre-serialize their own JSON string with no documentation of the expected shape. Retyped to Optional[Mapping[str, Any]]. Fixed in both sync and async
| api_version: Optional[str] = None, | ||
| credential_scopes: Optional[List[str]] = None, | ||
| extra_query: Optional[Mapping[str, str]] = None, | ||
| extra_headers: Optional[Mapping[str, str]] = None, |
There was a problem hiding this comment.
Darren Cohen (@dargilco) should extra headers and extra query part of kwargs for our standard?
Review feedback fixes:
- CSpell: add "redef" to sdk/ai/cspell.yaml (fixes CI failure on the
type: ignore[no-redef] comment in
sample_voice_agent_live_audio_conversation_async.py).
- PostEmitter.ps1: remove the _realtime.py regression guard -- verified
empirically (ran a real tsp-client update against the pinned commit; the
hand-written files came back byte-identical) that tsp-client update does
not touch files it doesn't generate, so the guard was unnecessary.
- pyproject.toml: rename the "realtime" optional-dependency extra to
"voice" (matches how it'll be documented for voice agents); update the
2 samples that referenced [realtime] to [voice].
- Samples: pin azure-ai-projects>=2.7.0 (drop the b1/--pre prerelease
pins -- this package no longer ships beta releases) across all 11 voice
samples; use the [voice] extra instead of manually listing aiohttp in
the 2 samples that need it.
- _realtime.py / aio/_realtime.py:
- Remove the foundry_features parameter from
RealtimeConnectionManager/Realtime.connect/AsyncRealtime.connect --
the realtime route is voice-agent-specific, so the header value is
always the same and callers should not need (or be able) to override
it. The value is now a hardcoded module constant.
- Fix structured_inputs' type: it was Optional[str], silently requiring
callers to pre-serialize their own JSON string with no documentation
of the expected shape. Retyped to Optional[Mapping[str, Any]],
matching the analogous generated model field
(CreateTelephonyCallJobRequest.structured_inputs: dict[str, any]),
and now serialized internally via SdkJSONEncoder.
- Updated test_realtime_client.py/_async.py's _make_manager() helpers
to match (foundry_features removed; the header-value assertions still
pass since it's now a module constant rather than a per-call override).
Test recordings:
- Generated live recordings for the 8 new voice-agent tests that had none
yet (test_voice_agent_crud(_async).py x6, test_read_conversation(_async)
x2), against a real Foundry project with a real gpt-realtime deployment.
Verified sanitization (endpoint/auth stripped) and re-verified end-to-end
in playback mode after a fresh assets restore from the new tag.
- Pushed the new recordings to Azure/azure-sdk-assets (tag
python/ai/azure-ai-projects_59c7584f68) via the GitHub Git Data REST API,
since a direct git push to that repo hangs in this environment; updated
assets.json accordingly.
Validated: black/pylint/mypy clean; full existing test suite unaffected
(740 passed, 49 skipped, 0 failed, up from 732 passed/8 failed).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Glenn Harper (glharper)
left a comment
Glenn Harper (glharper)
left a comment
There was a problem hiding this comment.
Comments 2-6 from the local review, checked against the current PR revision.
| if self._structured_inputs is not None: | ||
| headers["x-ms-voice-structured-inputs"] = json.dumps(self._structured_inputs, cls=SdkJSONEncoder) |
There was a problem hiding this comment.
2. [P2] Send structured inputs through the defined query parameter
connect(structured_inputs=...) serializes the mapping into an x-ms-voice-structured-inputs header, but the API contract defines a URL-encoded structured_input query parameter. Both the pinned TypeSpec route and the generated request builder specify that query parameter, not this header. Consequently, per-session instruction/greeting substitutions are not delivered through the defined service interface. The same issue exists in the async implementation.
Evidence: The original offline handshake captures showed no structured_input query entry, while the generated request builder given the same JSON produced that entry. The newer mapping-type change adds JSON serialization, but still writes only this header; the wire-contract mismatch remains.
Suggested fix: Serialize the mapping and add it to the query parameters under structured_input before URL encoding in both implementations. Keep the public parameter name if desired, but align its wire representation and handshake assertions with the generated contract.
There was a problem hiding this comment.
Good catch, confirmed against the generated request builder — fixed. structured_inputs is now serialized into the structured_input query parameter in both _realtime.py and aio/_realtime.py, matching the pinned TypeSpec route.
| while True: | ||
| try: | ||
| yield self.recv() | ||
| except ConnectionResetError: | ||
| return |
There was a problem hiding this comment.
3. [P2] Propagate abnormal WebSocket termination from event iterators
Both iterators catch every ConnectionResetError and return normally. The synchronous recv() converts all websockets.ConnectionClosed exceptions, including abnormal closure, to that exception. The asynchronous recv() uses it for both closure frames and WSMsgType.ERROR. Thus server failures and interrupted connections become indistinguishable from successful end-of-stream in the advertised for / async for interface. A caller can silently accept a truncated response, and its exception-based recovery path never runs. The same catch-and-return occurs in the async iterator.
Evidence: A synchronous ConnectionClosedError with close code 1011 caused list(connection) to return []. An asynchronous close-1011 frame and an asynchronous error frame carrying an OSError likewise ended iteration without raising.
Suggested fix: End iteration quietly only for normal closure. Preserve and propagate abnormal close codes and receive errors, including the underlying cause, in both implementations.
There was a problem hiding this comment.
Fixed in both implementations. recv() still raises ConnectionResetError for any closure, but _iter()/__aiter__ now only end iteration quietly for a graceful one — an abnormal close code (e.g. 1011) or a transport error now propagates instead of being silently swallowed. Added regression tests for both the graceful and abnormal paths (sync and async).
| if response_active: | ||
| await conn.response.cancel() | ||
| ap.skip_pending_audio() |
There was a problem hiding this comment.
4. [P2] Clear buffered speaker audio on barge-in after generation completes
On input_audio_buffer.speech_started, ap.skip_pending_audio() runs only while response_active is true. That flag becomes false on response.done, which marks the end of server generation, not the end of local playback. When the server generates audio faster than the speaker consumes it, speaking over the remaining buffered reply does not interrupt playback; the old reply continues over the user.
Evidence: Exercising the actual sample audio callback with a fake device, two seconds of queued PCM, and only 50 ms consumed, then delivering response.done followed by speech_started, left the next callback returning old non-silent audio.
Suggested fix: Clear pending local playback on every speech-start event. Guard only the server-side response.cancel() call with response_active; local playback and server generation need independent state.
There was a problem hiding this comment.
Confirmed and fixed — ap.skip_pending_audio() now runs unconditionally on speech_started, independent of response_active. Only the server-side response.cancel() call (and the "(listening...)" message) stay guarded by response_active, since cancelling with none active is a service error.
| except TimeoutError: | ||
| print("Timed out waiting for the agent's reply.") | ||
| conn.response.cancel() | ||
| return |
There was a problem hiding this comment.
5. [P2] Drain a cancelled response before accepting the next text turn
After a reply times out, the samples send response.cancel() and return to prompting without consuming the cancelled response's terminal event. The next turn's event pump returns on any RealtimeServerEventResponseDone, without matching the response ID. A late completion for the cancelled turn therefore ends the next turn's pump before the new reply is displayed, leaving the stream out of step with the user. The same issue exists in the async text sample.
Evidence: Deterministic sync and async runs supplied a first-turn timeout, the first response's cancelled response.done, and the second response's transcript/completion. Both issued two response requests but failed to display the second reply; the async run left its transcript and completion unread.
Suggested fix: Consume cancellation through the matching terminal event before reusing the stream, and correlate completion events with the active response ID. If cancellation cannot be confirmed within a bounded wait, end the session rather than accept another turn on an unsynchronized stream. Apply the same handling to the async greeting-timeout path.
There was a problem hiding this comment.
Fixed in both the sync and async samples (including the async greeting-timeout path). Each turn now tracks its response id from response.created, and after a timeout+cancel, a new bounded drain step consumes that response's terminal event (correlated by id) before the next turn starts — so a late completion can no longer be mistaken for the next turn's reply. If it can't be confirmed in time, the sample now ends the session cleanly instead of continuing on a stream it can't trust.
| except HttpResponseError as e: | ||
| if e.status_code == 404: | ||
| continue |
There was a problem hiding this comment.
6. [P2] Require a successful per-item audio retrieval in the conversation tests
Both tests tolerate every per-item audio request returning 404 without requiring a single successful audio retrieval. Their setup requests a stored audio response and requires a completed conversation with a merged recording, but a broken per-item audio route can still pass the test without ever reaching download_item_audio(). Recording that behavior would preserve the false-positive result in playback. The same issue exists in the async test.
Evidence: Running both unchanged test bodies with mocked completed conversations containing user and assistant-audio items passed despite two item-audio 404s and zero item-audio download calls in each run. This reproduction bypassed live setup and recording decorators; it did not contact the service.
Suggested fix: Continue tolerating 404 for genuinely non-audio items, but require at least one successful audio-item metadata retrieval and a nonempty download for Foundry-managed storage.
There was a problem hiding this comment.
Fixed in both the sync and async test — now tracks whether any item successfully retrieved audio and asserts that after the loop, so a fully-broken get_item_audio/download_item_audio route can no longer pass silently.
…nt samples to use create_version directly - _realtime.py / aio/_realtime.py: send structured_inputs as the documented structured_input query parameter instead of a custom header; distinguish graceful WebSocket closure from abnormal closure/transport errors so `for event in conn:` no longer silently swallows real failures. - sample_voice_agent_live_audio_conversation_async.py: always clear buffered local playback audio on barge-in, independent of whether a server response is still active. - sample_voice_agent_live_text_conversation.py / _async.py: drain a cancelled response's terminal event (correlated by response id) before starting the next turn, so a late completion can't be mistaken for the next reply. - test_voice_agent_conversations.py / _async.py: require at least one successful per-item audio retrieval instead of tolerating an all-404 run. - test_realtime_client.py / _async.py: add regression tests for the structured_input query parameter and the graceful-vs-abnormal closure distinction. - sample_voice_agent_live_audio_conversation_async.py, sample_voice_agent_live_text_conversation.py, sample_voice_agent_live_text_conversation_async.py: replace the generate()-then-patch-store two-step pattern with a single direct VoiceAgentDefinition(..., store=True) + create_version() call, matching sample_voice_agent_basic.py. Live-tested all three end-to-end. - GeneratePublicMethods.ps1 / docs/public-methods.md / api.md / api.metadata.yml: fix the generator to recognize the hand-written beta.realtime sub-client (a plain property, not a generated *Operations group) and regenerate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… into xitzhang/voice-agents-beta
4 participants
Summary
Adds Voice Agents to �zure-ai-projects, aligned with the .beta naming/structure already established on this branch ( eature/azure-ai-projects/vnext picked up the generated Voice Agents surface via #48950). This PR adds the pieces needed to make it a complete, usable feature: the hand-written realtime WebSocket client, samples, tests, and docs.
Related to #47803 (tracking �next -> main). cc Darren Cohen (@dargilco)
What's included
ealtime is a hand-written WebSocket entry point, not a generated REST operation, so it needs its own dedicated header test instead -- see est_realtime_client.py).
ealtime extra (websockets / �iohttp).
Renames applied while porting
This branch's TypeSpec commit turned out to be newer than the source content this was ported from, and had renamed several methods beyond just .beta nesting (verified against this branch's own generated code and docs/public-methods.md):
Validation