[TRTLLM-12670][feat] add /start_profile and /stop_profile endpoints to trtllm… by JunyiXu-nv · Pull Request #13872 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-05-08T00:03:35Z

…-serve

Expose iteration-scoped runtime profiling over HTTP on the OpenAI-compatible trtllm-serve endpoint, mirroring SGLang's developer-profiling interface.

POST /start_profile accepts output_dir, num_steps, start_step, activities (any of "CPU", "GPU", "CUDA_PROFILER"). With "CUDA_PROFILER" only, the server skips torch.profiler entirely so it composes cleanly with nsys profile -c cudaProfilerApi.
POST /stop_profile terminates the active window and flushes the trace.
PyExecutor.start_profile/stop_profile reuse the existing profile_start_iters / profile_stop_iters / cudaProfilerStart-Stop plumbing so env-var-based profiling behavior is preserved.
_profiler() now builds torch.profiler lazily on the first start iteration so runtime configuration takes precedence over the TLLM_TORCH_PROFILE_TRACE env var and consecutive start/stop windows work.
GenerationExecutor, BaseWorker, and GenerationExecutorRpcProxy proxy the new calls into the worker process; non-PyTorch backends fall through to a warning.
LLM.start_profile / stop_profile added as @set_api_status("prototype").
Smoke test covers default values, explicit overrides, and the stop path via mocked generator (no GPU required).

Dev Engineer Review

Added POST /start_profile and POST /stop_profile to trtllm-serve.
Added runtime profiling across LLMs, executors, workers, proxies, and PyExecutor.
Added iteration-scoped scheduling, lazy profiler creation, multi-rank traces, activity selection, and consecutive profiling windows.
Added input validation for CPU, GPU, and CUDA_PROFILER.
Added proxy-worker acknowledgements, timeout handling, and shutdown-state protection.
Added unsupported-backend warnings and profiling documentation.
Test-list entries use valid paths and cover endpoint smoke tests, executor profiling, and proxy synchronization.
CI results remain mixed. PR_Github #49393 succeeded, but multiple main pipelines failed.

QA Engineer Review

Added tests cover:

OpenAI server defaults, custom arguments, backend errors, and event-loop responsiveness.
PyExecutor scheduling, cancellation, validation, and clean failure state.
Proxy acknowledgements, error propagation, timeouts, serialization, thread ownership, and shutdown races.
test_trtllm_serve_profile_example.

The endpoint smoke test, test_profile_endpoints.py, and test_proxy_profile_sync.py are listed in tests/integration/test_lists/test-db/l0_a10.yml.

Verdict: sufficient.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

JunyiXu-nv · 2026-05-11T09:06:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-11T09:13:31Z

PR_Github #47712 [ run ] triggered by Bot. Commit: 86b3237 Link to invocation

tensorrt-cicd · 2026-05-11T19:55:51Z

PR_Github #47712 [ run ] completed with state SUCCESS. Commit: 86b3237
/LLM/main/L0_MergeRequest_PR pipeline #37608 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-13T14:00:17Z

/bot run

tensorrt-cicd · 2026-05-13T14:06:41Z

PR_Github #48187 [ run ] triggered by Bot. Commit: c5bc255 Link to invocation

tensorrt-cicd · 2026-05-13T15:23:53Z

PR_Github #48187 [ run ] completed with state SUCCESS. Commit: c5bc255
/LLM/main/L0_MergeRequest_PR pipeline #38007 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-14T03:38:49Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-14T03:44:34Z

PR_Github #48285 [ run ] triggered by Bot. Commit: 41eca47 Link to invocation

tensorrt-cicd · 2026-05-14T16:22:17Z

PR_Github #48285 [ run ] completed with state SUCCESS. Commit: 41eca47
/LLM/main/L0_MergeRequest_PR pipeline #38097 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…ints Two follow-up fixes for the CI failures introduced by the profile endpoints PR (NVIDIA#13872): 1. API stability reference: add start_profile / stop_profile entries to tests/unittest/api_stability/references/llm.yaml so TestLLM::test_{signature,docstring,modified_init,...} stops failing with "LLM has different methods: {'start_profile', 'stop_profile'}". 2. Ray WorkerExtension collision: allow a user-supplied WorkerExtension to override BaseWorker.start_profile / stop_profile. These names are explicitly designated extension points (the RLHF WorkerExtension in tensorrt_llm/llmapi/rlhf_utils.py uses torch.cuda.profiler directly), so _inject_worker_extension now copies overridable extension methods into the derived class dict, which makes the extension's impl win regardless of MRO order and keeps the conflict check strict for every other name. Fixes the cascade of "RayGPUWorker died during initialization / Worker class RayGPUWorker already defines 'start_profile'" errors across the Ray test suites (H100, H100_PCIe, B200). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>

The Pre-commit Check on PR NVIDIA#13872 failed with two issue families: * ruff-format (PEP-8 + 80-col yapf-equivalent) reformatted multiline asserts and f-strings in the new test files. * ruff-legacy reported 8 D205 regressions ('1 blank line required between summary line and description') vs the baseline. Some are in docstrings I added (proxy._wait_profile_ack, proxy.start_profile, proxy.stop_profile, test_e2e.test_trtllm_serve_profile_example, 4 docstrings in tests/unittest/executor/test_proxy_profile_sync.py) and a few are pre-existing violations whose line numbers shifted because of my additions (proxy.submit, test_e2e bench tests). Fix all 8 by inserting the required blank line after each docstring summary line and tightening summary lines so they are real sentences (D415 fix on the test_proxy_profile_sync.py module docstring). Also accept the ruff-format reflow on the two test files. Verified locally with: pre-commit run --files <touched files> — all hooks pass (yapf, ruff, ruff-format, ruff-legacy, autoflake, codespell, etc.). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-05-15T03:12:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-15T03:18:57Z

PR_Github #48498 [ run ] triggered by Bot. Commit: 941d851 Link to invocation

tensorrt-cicd · 2026-05-15T23:23:31Z

PR_Github #48498 [ run ] completed with state FAILURE. Commit: 941d851
/LLM/main/L0_MergeRequest_PR pipeline #38294 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tensorrt-cicd · 2026-08-27T17:45:58Z

PR_Github #69769 [ run ] completed with state ABORTED. Commit: f2edcd4

Link to invocation

tensorrt-cicd · 2026-08-27T19:32:39Z

PR_Github #69778 [ run ] completed with state SUCCESS. Commit: 60d0ac9
/LLM/main/L0_MergeRequest_PR pipeline #57074 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…o trtllm-serve Expose iteration-scoped runtime profiling over HTTP on the OpenAI-compatible trtllm-serve endpoint. - POST /start_profile accepts output_dir, num_steps, start_step, activities (any of "CPU", "GPU", "CUDA_PROFILER"). With "CUDA_PROFILER" only, the server skips torch.profiler entirely so it composes with `nsys profile -c cudaProfilerApi`. - POST /stop_profile terminates the active window and flushes the trace. - PyExecutor.start_profile/stop_profile reuse the existing profile_start_iters / profile_stop_iters / cudaProfilerStart-Stop plumbing so env-var-based profiling behavior is preserved. - GenerationExecutor, BaseWorker and GenerationExecutorRpcProxy proxy the calls into the worker process; non-PyTorch backends fall through to a warning. - LLM.start_profile / stop_profile added as @set_api_status("prototype"). Rebased onto main; the branch's 15 development commits are collapsed into one because several predate the tensorrt_llm/executor/ray_gpu_worker.py -> ray/gpu_worker.py move and could not be replayed onto current main without hand-porting each intermediate state. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

… before applying it `_sync_server_clock` estimates each ctx/gen server's clock offset with the NTP algorithm over a single HTTP round trip, then adds it to every perf-metric timestamp that server reports. That estimate's error is bounded by half the round-trip delay, because an asymmetric round trip is indistinguishable from a real clock offset -- and the handshake runs while the servers are still finishing startup, so a single sample regularly lands on a stalled event loop. On one host the true offset is exactly 0 (everything reads CLOCK_MONOTONIC), so whatever it computes is pure injected error. Across CI, 2394 of 2407 measured offsets are positive: the bias is systematic and lands entirely on the outbound leg, where DNS resolution, connection setup and the server's first-ever request are paid. `test_disaggregated_perf_metrics` asserts a 5 ms tolerance against that offset and has failed 4/4 runs here with 33-69 ms of injected skew, versus a 0.87 ms median across 128 baseline runs on main. - Warm up the connection first so DNS and connect costs stay out of the samples. - Probe up to `_CLOCK_SYNC_PROBES` times and keep the least-delayed round trip (NTP's clock filter), stopping early once a sample is conclusive so a healthy server still costs one probe. - Discard the offset when even the best delay exceeds `_CLOCK_SYNC_MAX_DELAY_SECONDS`: past that the estimate is worth less than the zero it would replace, since co-located servers share CLOCK_MONOTONIC and NTP-synced hosts agree to well under a millisecond. - Bound the cost: a dedicated per-request timeout instead of inheriting `_req_timeout_secs`, a wall-clock budget for the probe loop, and no retries against a server that errors -- servers are prepared sequentially, so an unresponsive one must not stall the queue behind it. Adds CPU-only unit tests driving the handshake against a stub endpoint with an injectable one-sided stall. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-08-27T19:36:50Z

Fix verified; one real regression found and fixed

The lazy ack channel works

Pipeline #57074, node a4u8g-0059.ipp2a2 — the same physical node where this PR previously measured 36/47 ms (build 55878) and 52/29 ms (54606):

	with eager queue (same node)	with lazy queue
handshake delay	36/47, 52/29 ms	2.12 / 1.90 ms
applied offset	~74 ms of pure error	0.17 / 0.048 ms
HMAC keys generated	8	6
causality assertion	failed, 3 attempts	passed, 1 attempt

The HMAC count is the load-bearing number: delays move with machine load, but the queue count is deterministic, so it confirms the lazy path is actually taken rather than the run getting lucky.

All three profile suites pass on the rewired ack channel — test_proxy_profile_sync 12 passed, test_profile_endpoints 12 passed, test_trtllm_serve_profile_example 17 passed. That last one drives /start_profile and /stop_profile through the HTTP layer against the real IPC proxy, so the synchronous "trace is on disk when this returns 200" guarantee survives the change.

The regression

The pipeline still failed, on something unrelated to the handshake:

AttributeError: PyExecutor object has no attribute _profile_enabled
tests/unittest/_torch/executor/test_kv_pool_rebalance.py::TestPpLoopDrainWiring  (4 tests, x86 and SBSA)

This PR wraps forward+sample in _step_scope(), which reads _profile_enabled. That fixture builds its executor with object.__new__(PyExecutor) and stubs "exactly the attributes one iteration touches" — so the new one has to be declared there. Fixed by adding exe._profile_enabled = False to _make_pp_loop_executor().

I took that over making _step_scope() defensive with getattr(..., False): production code should not loosen its own invariants to accommodate a test double, and that call is on the per-iteration path.

Checked the rest of the blast radius rather than assuming: ten test files construct PyExecutor via object.__new__, and only this one reaches _step_scope. test_py_executor.py uses the pattern 35 times across every executor loop and passes 149/149. The executor stage was 1062 passed / 4 failed, and those 4 are exactly the ones fixed here.

Head is now c29f879.

JunyiXu-nv · 2026-08-27T19:36:51Z

/bot run

tensorrt-cicd · 2026-08-27T19:42:40Z

PR_Github #69798 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-27T23:52:47Z

PR_Github #69798 [ run ] completed with state FAILURE. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57090 completed with status: 'UNSTABLE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JunyiXu-nv · 2026-08-27T23:58:18Z

Everything this PR touches is green; the one failure is unrelated flake

Pipeline #57090 came back UNSTABLE. Across all 4632 artifacts there is exactly one failing test:

A30-CPP-1 :: test_unit_tests[batch_manager-80]
subprocess.CalledProcessError: ctest ... returned non-zero exit status 8

That is a C++ ctest. This PR changes no C++ — git diff --name-only origin/main...HEAD matches zero .cpp/.cu/.h/.hpp/.cc files. And the test is broadly flaky right now: in the last 7 days it has failed 30 times across ~20 different PRs plus post-merge/main, of which this PR accounts for 1.

PR	failures (7d)
18033	2
post-merge / main	1
13872 (this PR)	1
10520, 16132, 16331, 16911, 17377, 17434, 17485, 17662, 17668, 17693, 17695, 17781, 17849, 17870, 17884, 17899, 17921, …	1 each

Everything this PR does touch passed, on a4u8g-0064.ipp2u2:

check	result
disagg handshake delay	1.23 / 1.47 ms (was 36/47 ms on this PR)
applied clock offset	0.15 / 0.26 ms (was ~74 ms of pure error)
HMAC keys generated	6 (was 8 — confirms the lazy path is taken)
`test_disaggregated_perf_metrics`	passed, first attempt, 0 assertions
executor unit tests (x86 / SBSA)	1066 / 1066 passed, 0 FAILED
`test_proxy_profile_sync`	12 passed
`test_profile_endpoints`	12 passed
`test_trtllm_serve_profile_example`	17 passed

The executor stage went 1062 passed / 4 failed → 1066 passed, which is exactly the four test_kv_pool_rebalance fixtures fixed in the last push. No test was retried anywhere in the pipeline, so nothing else failed and recovered either.

Re-running CI for the flaky C++ test.

JunyiXu-nv · 2026-08-27T23:58:19Z

/bot run

tensorrt-cicd · 2026-08-28T00:04:49Z

PR_Github #69830 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T02:48:01Z

PR_Github #69830 [ run ] completed with state SUCCESS. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57123 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-28T02:50:15Z

/bot run

JunyiXu-nv · 2026-08-28T02:51:40Z

Second run: different unrelated failure, and almost no coverage

Pipeline #57123 failed on one test, and it is not the same one as last time:

DGX_B200-8_GPUs-PyTorch-2 :: test_parallel_vae.py::TestParallelVAEDecode::test_decode_width_2gpu
torch.distributed.DistNetworkError: Failed to recv, got 0 bytes
TCP client failed to connect/validate to host localhost:32061 - timed out (try=19, timeout=600000ms)

CI's own analysis concluded "PR likely to blame?: No", but I checked rather than taking that on faith, and the honest picture is more nuanced than for the previous flake:

test_decode_width_2gpu ran 233 times in the last 7 days: 229 passed, 3 skipped, 1 failed — and that 1 is this run. So it is rare (0.4%), not broadly flaky the way batch_manager is (30 failures across ~20 PRs in the same window).
What makes it still unrelated is the shape, not the rate: it is a torch.distributed TCPStore rendezvous timeout (600 s, 19 retries) in a VisualGen multi-GPU VAE test. This PR touches no VisualGen code, no distributed init, and no C++ — there is no causal path from a lazily-created IPC queue in trtllm-serve to a TCPStore handshake in a separate B200 job.

Worth flagging: this run produced almost no coverage. 268 artifacts against 4632 in #57090, and zero test_disaggregated_perf_metrics artifacts — the B200 failure tripped fail-fast and aborted the single-GPU stages before they ran. So #57123 neither confirms nor contradicts anything about this PR.

The run that actually exercised this PR is still #57090: handshake 1.23/1.47 ms, HMAC keys 6, disagg test passed first attempt, executor unit tests 1066/1066 on both architectures, and all three profile suites green (12/17/12). Its only failure was the batch_manager C++ ctest, which this PR cannot touch (zero C++ files changed) and which failed 30 times across ~20 other PRs plus main in the same week.

Re-running.

JunyiXu-nv · 2026-08-28T02:51:41Z

/bot run

tensorrt-cicd · 2026-08-28T02:56:29Z

PR_Github #69859 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T02:58:41Z

PR_Github #69860 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T03:02:22Z

PR_Github #69859 [ run ] completed with state ABORTED. Commit: c29f879

Link to invocation

tensorrt-cicd · 2026-08-28T04:44:51Z

PR_Github #69860 [ run ] completed with state SUCCESS. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57150 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-28T04:46:45Z

/bot run

JunyiXu-nv · 2026-08-28T04:47:36Z

Three runs, three different unrelated failures

run	pipeline	artifacts	failure	that test's 7-day record
1	#57090	4632 (complete)	`batch_manager` C++ ctest	30 failures across ~20 PRs + main
2	#57123	268 (aborted early)	VAE B200 TCPStore timeout	1 failure / 233 runs
3	#57150	188 (aborted early)	UCX/MPI endpoint timeout, `test_gen_first[adp-mtp2]`	14 failures / 453 runs, across 7 sources — 5 of them on post-merge main

Run 3 is the one worth spelling out, because CI rated it "Confidence: Medium — the PR does touch disaggregated serving". The failure is ucx send failed: Endpoint timeout → PMIX ERROR: UNREACHABLE → MPI_ABORT on a Slurm H100 node: the KV-cache transport layer between MPI ranks. This PR adds two HTTP endpoints, makes one IPC queue lazy, and changes a clock handshake — none of that reaches UCX or MPI. And the same test failed 5 times on main in the same week, where this PR does not exist.

Runs 2 and 3 also carry no evidence either way about this PR: fail-fast aborted them before the single-GPU stages ran, so neither produced a single test_disaggregated_perf_metrics artifact (268 and 188 artifacts, against 4632 in run 1).

Run #57090 remains the only complete exercise of this PR, and everything it touches passed:

check	result
disagg handshake delay	1.23 / 1.47 ms (was 36/47 ms with the eager queue, same node class)
applied clock offset	0.15 / 0.26 ms (was ~74 ms of pure error)
HMAC keys generated	6 (was 8 — confirms the lazy path)
`test_disaggregated_perf_metrics`	passed, first attempt, 0 assertions
executor unit tests (x86 / SBSA)	1066 / 1066 passed, 0 FAILED
`test_proxy_profile_sync`	12 passed
`test_profile_endpoints`	12 passed
`test_trtllm_serve_profile_example`	17 passed

Re-running once more.

JunyiXu-nv · 2026-08-28T04:47:37Z

/bot run

tensorrt-cicd · 2026-08-28T04:52:14Z

PR_Github #69881 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T04:53:38Z

PR_Github #69883 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T04:59:52Z

PR_Github/13872-c29f879 #69881 was force-killed by a newer pipeline run.
L0 job information not available (job may not have been triggered yet).

Link to superseding invocation

tensorrt-cicd · 2026-08-28T06:34:25Z

PR_Github #69883 [ run ] completed with state SUCCESS. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-08-28T06:37:01Z

/bot run

JunyiXu-nv · 2026-08-28T06:37:34Z

Pausing re-runs: the RDMA/UCX fabric is unhealthy cluster-wide

Four runs, four different failures, none of them this PR — and the last two are the same class:

run	pipeline	artifacts	failure
1	#57090	4632 (complete)	`batch_manager` C++ ctest
2	#57123	268	VAE B200 TCPStore rendezvous timeout
3	#57150	188	`ucx send failed: Endpoint timeout` → `MPI_ABORT`
4	#57168	117	`ibv_create_ah ... Connection timed out` → `NIXL_ERR_NOT_FOUND`

Runs 3 and 4 are both UCX/RDMA transport failures on DGX H100 multi-GPU disagg stages, so I checked whether that is specific to this PR. It is not.

test_disaggregated_serving has failed 40 times in the last 24 hours across 11 different PRs plus post-merge main, concentrated on exactly those stages:

DGX_H100-4_GPUs-PyTorch-Others-2     10
DGX_H100-4_GPUs-PyTorch-DeepSeek-1    8
DGX_H100-4_GPUs-PyTorch-Others-1      8
DGX_H100-2_GPUs-PyTorch-Others-1      6
DGX_H100-2_GPUs-PyTorch-Others-2      5

This PR accounts for 5 of those 40. Re-running will not converge until the fabric is healthy, so I have stopped re-triggering rather than keep burning CI.

Runs 2-4 also carry no evidence about this PR either way: fail-fast aborted them before the single-GPU stages ran — 117-268 artifacts against 4632, and zero test_disaggregated_perf_metrics artifacts in any of them.

Run #57090 remains the only complete exercise of this PR, and everything it touches passed:

check	result
disagg handshake delay	1.23 / 1.47 ms (was 36/47 ms with the eager queue, same node class)
applied clock offset	0.15 / 0.26 ms (was ~74 ms of pure error)
HMAC keys generated	6 (was 8 — confirms the lazy path is taken)
`test_disaggregated_perf_metrics`	passed, first attempt, 0 assertions
executor unit tests (x86 / SBSA)	1066 / 1066 passed, 0 FAILED
`test_proxy_profile_sync`	12 passed
`test_profile_endpoints`	12 passed
`test_trtllm_serve_profile_example`	17 passed

Suggest waiting for the RDMA issue to clear before re-running.

tensorrt-cicd · 2026-08-28T06:42:34Z

PR_Github #69910 [ run ] triggered by Bot. Commit: c29f879 Link to invocation

tensorrt-cicd · 2026-08-28T08:26:01Z

PR_Github #69910 [ run ] completed with state FAILURE. Commit: c29f879
/LLM/main/L0_MergeRequest_PR pipeline #57196 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned JunyiXu-nv May 8, 2026

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 69b3d68 to 86b3237 Compare May 11, 2026 09:05

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 0b70c55 to 029a94b Compare May 11, 2026 09:20

JunyiXu-nv changed the title ~~[None][feat] add /start_profile and /stop_profile endpoints to trtllm…~~ [TRTLLM-12670][feat] add /start_profile and /stop_profile endpoints to trtllm… May 14, 2026

JunyiXu-nv added the api-compatible Accepted LLM API contract change that is backwards-compatible label May 14, 2026

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch from a07a3da to 41eca47 Compare May 14, 2026 03:34

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch from 41eca47 to fcb8426 Compare May 15, 2026 03:03

JunyiXu-nv force-pushed the feat/trtllm-serve-profile-endpoints branch 2 times, most recently from 1e41133 to 941d851 Compare May 15, 2026 03:10

JunyiXu-nv added 2 commits August 27, 2026 19:36

Uh oh!

Conversation

JunyiXu-nv commented May 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

JunyiXu-nv commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

JunyiXu-nv commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

JunyiXu-nv commented May 14, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

JunyiXu-nv commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented May 15, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

JunyiXu-nv commented Aug 27, 2026

Fix verified; one real regression found and fixed

The lazy ack channel works

The regression

Uh oh!

JunyiXu-nv commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 27, 2026

Uh oh!

JunyiXu-nv commented Aug 27, 2026

Everything this PR touches is green; the one failure is unrelated flake

Uh oh!

JunyiXu-nv commented Aug 27, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Second run: different unrelated failure, and almost no coverage

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

tensorrt-cicd commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Uh oh!

JunyiXu-nv commented Aug 28, 2026

Three runs, three different unrelated failures

Uh oh!

JunyiXu-nv commented Aug 28, 2026

JunyiXu-nv commented May 8, 2026 •

edited by coderabbitai Bot

Loading