[agentserver] Make per-request span flush non-blocking - #48955
[agentserver] Make per-request span flush non-blocking#48955Harsheet Shah (harsheet-shah) wants to merge 2 commits into
Conversation
The Responses endpoint flushed spans synchronously in the request `finally` block via `flush_spans()`, which runs `TracerProvider.force_flush` inline. On the async handler this blocks the asyncio event loop until the exporter drains, serialising concurrent requests behind a single export (head-of-line blocking) and adding the export time to every response. Add two helpers to azure-ai-agentserver-core: - `flush_spans_async`: offloads the blocking force_flush to a worker thread so it never stalls the event loop (same durability guarantee). - `schedule_flush_spans`: fire-and-forget flush that returns immediately so the response is not delayed (requires a platform drain window before freeze). The Responses hot-path flush now dispatches on `AGENTSERVER_FLUSH_MODE`: `async` (default, off the event loop), `background` (respond first), or `sync` (legacy). Default `async` removes event-loop head-of-line blocking with no change to durability. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 93f793d2-1b86-4677-908c-3722d4fef290
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
The dependency floor and API snapshot are stale, background flushing is unbounded, and the new behavior lacks tests.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds non-blocking span flushing to prevent synchronous telemetry export from blocking async requests.
Changes:
- Adds asynchronous and background flush helpers.
- Introduces configurable per-request flush modes.
- Documents the behavior changes.
File summaries
| File | Description |
|---|---|
| core/CHANGELOG.md | Documents new tracing helpers. |
| core/_tracing.py | Implements async and background flushing. |
| core/init.py | Exports the new public helpers. |
| responses/CHANGELOG.md | Documents configurable flush modes. |
| responses/_endpoint_handler.py | Selects the flush strategy per request. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 5
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| from azure.ai.agentserver.core import ( # pylint: disable=import-error,no-name-in-module | ||
| FoundryAgentRequestContext, | ||
| flush_spans, | ||
| flush_spans_async, |
| "flush_spans_async", | ||
| "schedule_flush_spans", |
| return | ||
| try: | ||
| loop = asyncio.get_running_loop() | ||
| await loop.run_in_executor(None, flush, timeout_millis) |
| # latency, but requires the platform to grant | ||
| # a brief drain window before freezing. | ||
| # "sync" -> flush_spans(): legacy blocking behaviour. | ||
| _flush_mode = os.environ.get("AGENTSERVER_FLUSH_MODE", "async").lower() |
This comment has been minimized.
This comment has been minimized.
- Coalesce background flushing: at most one flush task runs at a time; concurrent requests collapse into a single follow-up pass instead of spawning a retained task per request (bounded under load). - Raise azure-ai-agentserver-core floor to >=2.2.0b2 in responses, since the handler now imports flush_spans_async/schedule_flush_spans (added in b2). - Record flush_spans_async/schedule_flush_spans in core api.md. - Extract _flush_spans_for_mode dispatch helper (case/whitespace-insensitive, unknown values fall back to the async default). - Add tests: flush_spans_async non-blocking/timeout/exception/no-op; schedule_flush_spans coalescing + sync fallback; handler flush-mode dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 93f793d2-1b86-4677-908c-3722d4fef290
|
Thanks for the review — addressed in 8fe884a: 1. Unbounded background flushing (_tracing.py): schedule_flush_spans now runs a single coalesced flush. At most one background flush task exists at a time; requests that arrive while a flush is in flight set a pending flag so the running task performs exactly one follow-up pass afterward (capturing spans produced during the active flush). No per-request task/executor-queue growth, and no redundant concurrent orce_flush calls. Access is confined to the event-loop thread, so no lock is needed. Covered by 2. Stale core dependency floor ( 3. api.md missing exports: added 4/5. Missing tests:
All 17 new tests pass locally. |
There was a problem hiding this comment.
🟡 Changes recommended
Streaming spans are flushed before stream execution, and the API metadata hash is stale.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 2
- Review effort level: Balanced
| await _flush_spans_for_mode( | ||
| os.environ.get(_FLUSH_MODE_ENV, _DEFAULT_FLUSH_MODE) | ||
| ) |
| def azure.ai.agentserver.core.set_request_context(context: FoundryAgentRequestContext) -> Token[FoundryAgentRequestContext]: ... | ||
|
|
||
|
|
||
| async def azure.ai.agentserver.core.flush_spans_async:async(timeout_millis: int = 5000) -> None: ... |
[Pilot] PR Pipeline Failure AnalysisWhat failedAzure Pipelines build 6809423 failed on 1. CSpell (validation) — Build Analyze job 2. Pylint (validation) — Build Analyze job 3. E2E test failure — Build Test ubuntu2404_312 Relevant pipeline outputRecommended next steps
Automated fix: Fix found, view and apply fix
|
Summary
The Responses endpoint flushes spans synchronously in the request
finallyblock viaflush_spans(), which runsTracerProvider.force_flushinline. On the async handler this blocks the asyncio event loop until the exporter drains, so under concurrency every request is serialised behind a single export (head-of-line blocking), and the export time is added to every response.Changes
azure-ai-agentserver-core— two new public helpers in_tracing.py:flush_spans_async()— offloads the blockingforce_flushto a worker thread (run_in_executor) so it never stalls the event loop. Same durability guarantee asflush_spans().schedule_flush_spans()— fire-and-forget flush that returns immediately so the response is not delayed. Retains a strong task reference; falls back to sync flush when no loop is running. Safe only when the platform grants a drain window before freezing.azure-ai-agentserver-responses— the hot-path flush now dispatches onAGENTSERVER_FLUSH_MODE:async(default) —await flush_spans_async(); off the event loop, identical durability, removes head-of-line blocking.background—schedule_flush_spans(); respond first, flush in the background.sync— legacy blockingflush_spans().Measured impact
Deployed on a Foundry hosted MAF agent (uksouth, Responses protocol), 15 warm requests each, same image, only
AGENTSERVER_FLUSH_MODEvaried:sync(baseline)background≈ 0.8–1.3 s / request (~10–16%) off the response path even with light instrumentation; larger under heavy instrumentation (bigger span exports). Default
asyncmode removes event-loop head-of-line blocking with no durability change.Backwards compatibility
Default behaviour is
async(still awaited, just off the event loop) — no span loss vs. today.syncrestores the exact prior behaviour.