[https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity by chienchunhung · Pull Request #17137 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-31T19:21:20Z

Summary

This PR updates one GB300 DeepSeek V4 Pro disaggregated perf-sanity target:

explicitly enables cache_transceiver_precheck;
sets GEN and CTX kv_transfer_timeout_ms from 600000 ms to 60000 ms;
removes the exact gen-only test's NVBUG #6601537 waiver so the modified target is exercised.

PR #17223 has merged into main. The current PR head b5a37138566de927bce855e7c841d53e4b417eeb is rebased on main at 8d0e9adb36c45db6bb2729b598235080bd0ee5be, which includes both #17223 and the checkpoint-prefetch OOM fix #17430.

Validation

Targeted command:

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

Targeted test:

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen-only-gb300_deepseek-v4-pro-fp4_8k1k_con180_ctx3_dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL]

Fresh unwaived validation is pending at commit b5a371385: trigger comment. Test reuse is disabled.

Previous L0 #54047 validated the cache-transceiver precheck, but the exact E2E test was still waived and skipped (0 passed, 1 skipped). It is not an end-to-end validation result.

Historical L0 #51530 passed the exact smaller-scale test and completed 180/180 requests with the 60-second timeout, but predates the final #17223 revision and #17430.

This 3-CTX, concurrency-180 proxy does not replace the original 8-CTX, concurrency-1760 AgentPerf validation.

Related PRs

[https://nvbugs/6480621][fix] Preserve KV ownership in disaggregated precheck #17223: merged ownership/deadline fix.
[https://nvbugs/6272397][fix] Prevent host OOM during checkpoint prefetch #17430: merged checkpoint-prefetch host-OOM fix; included in the current base.
[None][feat] cache transceiver test in Perf sanity #16674: introduced the precheck being exercised.

chienchunhung · 2026-07-31T19:21:53Z

/bot run --disable-fail-fast --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

chienchunhung · 2026-07-31T19:25:04Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-07-31T19:27:43Z

PR_Github #63105 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T19:31:38Z

PR_Github #63106 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T19:35:23Z

PR_Github #63105 [ run ] completed with state ABORTED. Commit: 38ca970

Link to invocation

tensorrt-cicd · 2026-07-31T21:00:54Z

PR_Github #63106 [ run ] completed with state SUCCESS. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51197 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-07-31T23:00:28Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-07-31T23:06:22Z

PR_Github #63144 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-07-31T23:13:50Z

PR_Github #63144 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51231 (Partly Tested) completed with status: 'ABORTED'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chienchunhung · 2026-07-31T23:35:35Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-07-31T23:41:53Z

PR_Github #63150 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-08-01T00:38:05Z

PR_Github #63150 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51234 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-01T01:22:37Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-08-01T01:28:42Z

PR_Github #63165 [ run ] triggered by Bot. Commit: 38ca970 Link to invocation

tensorrt-cicd · 2026-08-01T02:53:55Z

PR_Github #63165 [ run ] completed with state FAILURE. Commit: 38ca970
/LLM/main/L0_MergeRequest_PR pipeline #51249 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-03T20:38:32Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-1"

tensorrt-cicd · 2026-08-03T20:44:14Z

PR_Github #63555 [ run ] triggered by Bot. Commit: 3e6c120 Link to invocation

tensorrt-cicd · 2026-08-03T21:27:26Z

PR_Github #63555 [ run ] completed with state SUCCESS. Commit: 3e6c120
/LLM/main/L0_MergeRequest_PR pipeline #51521 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-03T21:42:12Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-03T21:47:58Z

PR_Github #63564 [ run ] triggered by Bot. Commit: b7bd98a Link to invocation

tensorrt-cicd · 2026-08-03T22:59:04Z

PR_Github #63564 [ run ] completed with state SUCCESS. Commit: b7bd98a
/LLM/main/L0_MergeRequest_PR pipeline #51530 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

tensorrt-cicd · 2026-08-10T21:37:50Z

PR_Github #65119 [ run ] completed with state FAILURE. Commit: d0a7c6e
/LLM/main/L0_MergeRequest_PR pipeline #52916 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

chienchunhung · 2026-08-11T17:10:35Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

chienchunhung · 2026-08-11T18:26:22Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-11T18:32:48Z

PR_Github #65363 [ run ] triggered by Bot. Commit: 51841d8 Link to invocation

tensorrt-cicd · 2026-08-11T20:42:23Z

PR_Github #65363 [ run ] completed with state FAILURE. Commit: 51841d8
/LLM/main/L0_MergeRequest_PR pipeline #53128 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-11T22:42:15Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-11T22:50:05Z

PR_Github #65410 [ run ] triggered by Bot. Commit: 51841d8 Link to invocation

tensorrt-cicd · 2026-08-12T02:17:57Z

PR_Github #65410 [ run ] completed with state FAILURE. Commit: 51841d8
/LLM/main/L0_MergeRequest_PR pipeline #53167 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-15T00:33:12Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-15T00:38:41Z

PR_Github #66404 [ run ] triggered by Bot. Commit: de31862 Link to invocation

tensorrt-cicd · 2026-08-15T01:29:51Z

PR_Github #66404 [ run ] completed with state SUCCESS. Commit: de31862
/LLM/main/L0_MergeRequest_PR pipeline #54047 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chienchunhung · 2026-08-19T17:09:13Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-19T17:18:13Z

PR_Github #67509 [ run ] triggered by Bot. Commit: c32c4e4 Link to invocation

chienchunhung · 2026-08-19T17:19:08Z

/bot run --disable-fail-fast --disable-reuse-test --stage-list "GB300-44_GPUs-11_Nodes-PyTorch-Disagg-PerfSanity-CTX3-NODE1-GPU4-GEN1-NODE8-GPU32-Post-Merge-2"

tensorrt-cicd · 2026-08-19T17:31:39Z

PR_Github #67513 [ run ] triggered by Bot. Commit: b5a3713 Link to invocation

tensorrt-cicd · 2026-08-19T17:36:38Z

PR_Github #67509 [ run ] completed with state ABORTED. Commit: c32c4e4

Link to invocation

tensorrt-cicd · 2026-08-19T19:45:20Z

PR_Github #67513 [ run ] completed with state FAILURE. Commit: b5a3713
/LLM/main/L0_MergeRequest_PR pipeline #55006 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-08-20T22:10:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-20T22:18:47Z

PR_Github #67986 [ run ] triggered by Bot. Commit: 6573ef3 Link to invocation

tensorrt-cicd · 2026-08-21T01:06:20Z

PR_Github #67986 [ run ] completed with state SUCCESS. Commit: 6573ef3
/LLM/main/L0_MergeRequest_PR pipeline #55440 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned chienchunhung Jul 31, 2026

chienchunhung changed the title ~~[https://nvbugs/6480621][test] Validate 60-second KV transfer timeout~~ [https://nvbugs/6480621][test] Revert to 60-second KV transfer timeout for GB300 DeepSeek V4 Pro disaggregated perf-sanity Jul 31, 2026

Shixiaowei02 approved these changes Aug 3, 2026

View reviewed changes

chienchunhung mentioned this pull request Aug 3, 2026

[https://nvbugs/6480621][fix] Preserve KV ownership in disaggregated precheck #17223

Merged

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 38ca970 to 3e6c120 Compare August 3, 2026 20:37

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 3e6c120 to b7bd98a Compare August 3, 2026 21:40

chienchunhung marked this pull request as ready for review August 3, 2026 23:41

chienchunhung requested review from a team as code owners August 3, 2026 23:41

Shixiaowei02 added the ci: full pre-merge approved label Aug 11, 2026

nv-xtf approved these changes Aug 11, 2026

View reviewed changes

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from d0a7c6e to 51841d8 Compare August 11, 2026 18:25

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 51841d8 to 3fa575a Compare August 12, 2026 18:32

BowenFu approved these changes Aug 13, 2026

View reviewed changes

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from 3fa575a to de31862 Compare August 15, 2026 00:32

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from c32c4e4 to b5a3713 Compare August 19, 2026 17:18

chienchunhung added 2 commits August 20, 2026 15:08

[https://nvbugs/6480621][test] Validate 60-second KV transfer timeout

44748aa

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

[https://nvbugs/6480621][test] Unwaive GB300 DeepSeek V4 validation

6573ef3

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the codex/nvbug-6480621-timeout-60s-ci branch from b5a3713 to 6573ef3 Compare August 20, 2026 22:10

Uh oh!

Conversation

chienchunhung commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Related PRs

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

chienchunhung commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

chienchunhung commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

tensorrt-cicd commented Aug 1, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

chienchunhung commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 10, 2026

Uh oh!

chienchunhung commented Aug 11, 2026

Uh oh!

chienchunhung commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

chienchunhung commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 12, 2026

Uh oh!

chienchunhung commented Aug 15, 2026

Uh oh!

tensorrt-cicd commented Aug 15, 2026

Uh oh!

tensorrt-cicd commented Aug 15, 2026

Uh oh!

chienchunhung commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

chienchunhung commented Aug 19, 2026

Uh oh!

tensorrt-cicd commented Aug 19, 2026

Uh oh!

chienchunhung commented Jul 31, 2026 •

edited

Loading