[https://nvbugs/6590664][fix] Reap idle single-rank CTX transfers

[chienchunhung]

Summary

• Reap completed context-side transfers when a single-rank CTX Python executor is idle in synchronous-transfer mode.
• Release the KV blocks pinned by completed sender sessions so CTX can continue admitting requests.
• Remove the waiver for the exact GB200 GPT-OSS disaggregated gen_only perf-sanity test.

Root cause

The affected test uses CTX TP1 and GEN TP4 with TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1.

In synchronous-transfer mode, _check_disagg_transfer_progress_when_idle() returned at the multi-rank safety guard before polling check_context_transfer_status(). GEN-side KV receives could complete, but the corresponding CTX sender sessions were not reaped. Consequently, AsyncTransferManager.end_transfer() was not reached, completed transfers kept their KV blocks pinned, and CTX eventually stopped admitting requests with its KV pool at 99.9% utilization.

The apparent memory pressure and stalled GEN queue were downstream symptoms of missing CTX transfer cleanup—not a device OOM or a GEN model-forward failure.

Fix

When all of the following are true, poll CTX transfer status with timeout 0 before returning:

• the executor is idle;
• synchronous GEN transfer is configured;
• the executor has one rank; and
• at least one transfer is in flight.

This is the minimal liveness fix needed by the affected CTX TP1 topology. It is extracted from the broader work in #16402 without including that PR's queue-wakeup, timeout, cancellation, request-broadcast, C++, or client-lifecycle changes.

Safety boundary

The existing guard for multi-rank synchronous transfer remains intact.

A synchronous GEN receive is blocking and rank-local: one model-parallel rank may still be in request_and_receive_sync() while another reaches the idle-progress path. Context-status progress may enter model-parallel collectives, so polling it from only the idle rank can deadlock.

Asynchronous GEN transfer already uses the rank-coordinated idle-progress path added by #17107. The remaining generalization—multi-rank CTX with synchronous GEN transfer—requires explicitly coordinated polling such as the broader queue/broadcast machinery in #16402. It is intentionally out of scope because this test uses CTX TP1.

Targeted regression

Test

perf/test_perf_sanity.py::test_e2e[disagg_upload-gen_only-gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL]

CI stage

GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1

The test remains registered in tests/integration/test_lists/test-db/l0_gb200_multi_nodes_perf_sanity_ctx1_node1_gpu1_gen1_node1_gpu4.yml; this PR removes only its waiver from waives.txt.

Validation

• Added a focused unit regression proving that idle, single-rank CTX reaps synchronous transfers without entering distributed collectives.
• Preserved the existing multi-rank regression proving synchronous GEN does not enter idle-progress collectives.
• Repository pre-commit checks passed on all modified files.
• Targeted CI passed on rebased head 53bbe3cecf6e9f10574327865f0ab88ab2d5fe44:
  • PR_Github #66052 and L0_MergeRequest_PR #53735 completed successfully.
  • The CI report recorded 1 passed, 0 failed, and 0 skipped tests.
  • OpenSearch recorded the exact GPT-OSS selector above as PASSED—not waived or skipped—in 410.013 seconds on GB200 Slurm job 6131689.
  • This directly validates that the NVBUG 6590664 reproducer completes successfully with the fix. Because the case has historically been intermittent, additional full-CI results provide further confidence against recurrence.
• Current-head targeted CI passed after rebasing onto main with [None][chore] Scope TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP to the gen worker in disagg gen_only #17535 on a3f6d8f000288f9aa62696719af34c9eb673ea5b:
  • PR_Github #67566, L0_MergeRequest_PR #55057, and L0_Test-SBSA-Multi-GPU #3375 completed successfully.
  • The CI report recorded 1 passed, 0 failed, and 0 skipped tests in the ...Post-Merge-1 stage.
  • The raw child-job log shows reuse_test=false, selects only the exact GPT-OSS selector above, and records it as PASSED in 369.39 seconds (1 passed; OpenSearch duration 370.310 seconds).
  • GB200 Slurm job 6337974 completed with exit code 0 on nvl72167-T[14,16] (880-second Slurm runtime).
  • Because [None][chore] Scope TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP to the gen worker in disagg gen_only #17535 switches CTX workers to asynchronous transfer, this current-head result validates that the exact NVBUG 6590664 test is safe to unwaive on the current async path; it does not exercise this PR's synchronous idle-reaping branch. The earlier 53bbe3ce run directly validates that synchronous path.

Related bugs: NVBUG 6590664, parent NVBUG 6581075.

Dev Engineer Review

• Removed the GB200 GPT-OSS 120B disaggregated gen_only performance-sanity waiver.
• Updated py_executor.py to poll and reap completed synchronous context transfers for idle single-rank CTX executors.
• Released transfers allow pinned KV blocks to be reused before later context scheduling.
• Preserved the multi-rank synchronous-transfer safety guard.
• Added no public API changes.
• The change is small and consistent with the existing control flow.
• Targeted validation passed after rebasing.
• PR #17535 changed CTX workers to asynchronous transfer. The unwaived performance test may not exercise this synchronous-transfer fix directly.
• The fix remains useful for synchronous-transfer paths.

QA Engineer Review

• Added a regression test in tests/unittest/_torch/executor/test_py_executor.py.
• The test verifies that idle single-rank synchronous context-transfer mode polls transfer status while an asynchronous transfer is in flight.
• The test skips distributed collectives and generation-transfer polling.
• Removed one entry from tests/integration/test_lists/waives.txt for the GB200 GPT-OSS 120B disaggregated generation configuration.
• No test-db/ or qa/ files were modified.
• The regression test is not listed in test-db/ or qa/.
• Verdict: needs follow-up. Confirm CI coverage for the regression test and confirm whether the unwaived performance test exercises the intended synchronous-transfer path.

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49f8b563-64ca-499e-ad08-451716ba08ae

📥 Commits

Reviewing files that changed from the base of the PR and between e189237 and 76abf35.

📒 Files selected for processing (3)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tests/integration/test_lists/waives.txt
• tests/unittest/_torch/executor/test_py_executor.py

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

🚧 Files skipped from review as they are similar to previous changes (2)

• tests/unittest/_torch/executor/test_py_executor.py
• tensorrt_llm/_torch/pyexecutor/py_executor.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The change adds nonblocking context-transfer reaping during single-rank synchronous idle progress, adds regression coverage, and removes the related GB200 GPT-OSS 120B performance-test waiver.

Changes

Disaggregated transfer execution

Layer / File(s)	Summary
Reap idle context transfers tensorrt_llm/_torch/pyexecutor/py_executor.py, tests/unittest/_torch/executor/test_py_executor.py	The single-rank idle path checks and reaps completed context transfers without blocking. The regression test verifies context-transfer polling and excludes distributed collectives and generation-transfer polling.
Remove obsolete performance waiver tests/integration/test_lists/waives.txt	The skip entry for the GB200 GPT-OSS 120B disaggregated generation performance test was removed.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 76abf

The PR adds localized cleanup for completed single-rank context transfers and removes the corresponding test waiver; targeted validation passed, and no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: pcastongay, lori-ren, shixiaowei02, tongyuantongyu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required ticket/type format and clearly summarizes the synchronous single-rank context-transfer fix.
Description check	✅ Passed	The description covers the issue, root cause, fix, safety boundary, regression test, waiver removal, and validation results.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #65631 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65631 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53353 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #65652 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65652 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53378 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #65737 [ run ] triggered by Bot. Commit: a785da6 Link to invocation

[tensorrt-cicd]

PR_Github #65737 [ run ] completed with state FAILURE. Commit: a785da6
/LLM/main/L0_MergeRequest_PR pipeline #53453 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #66052 [ run ] triggered by Bot. Commit: 53bbe3c Link to invocation

[tensorrt-cicd]

PR_Github #66052 [ run ] completed with state SUCCESS. Commit: 53bbe3c
/LLM/main/L0_MergeRequest_PR pipeline #53735 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-reuse-test

[tensorrt-cicd]

PR_Github #66312 [ run ] triggered by Bot. Commit: 53bbe3c Link to invocation

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[chienchunhung]

#17535 is this PR related to the change? TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1 has not been exported in ctx worker in perf sanity test,so the change may not be exercised by the unwaived perf test.

@chuangz0 Thanks for bringing this to my awareness; confirmed that PR#17535 switched to use async transfer for CTX, which makes the current fixes in this PR non-exercised for the targeted test anymore.

The validation evidence in this PR was obtained before including PR#17535's changes, so IMO the fixes are still valid (i.e., once the sync path is explicitly configured). So my plan is to:

• Keep the fixes
• Rebase to the latest main and pick up the changes introduced by PR#17535
• Verify again that the targeted test can pass (or fail due to other reason). If the test can pass, the test un-waiving is still valid.

[chienchunhung]

/bot run --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #67135 [ run ] triggered by Bot. Commit: a3f6d8f Link to invocation

[chienchunhung]

/bot run --post-merge --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #67141 [ run ] triggered by Bot. Commit: a3f6d8f Link to invocation

[tensorrt-cicd]

PR_Github #67141 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

[tensorrt-cicd]

PR_Github #67135 [ run ] completed with state SUCCESS. Commit: a3f6d8f
/LLM/main/L0_MergeRequest_PR pipeline #54671 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --post-merge --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-2"

[tensorrt-cicd]

PR_Github #67508 [ run ] triggered by Bot. Commit: a3f6d8f Link to invocation

[tensorrt-cicd]

PR_Github #67508 [ run ] completed with state SUCCESS. Commit: a3f6d8f
/LLM/main/L0_MergeRequest_PR pipeline #55001 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --post-merge --disable-reuse-test --disable-fail-fast --stage-list "GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-1"

[tensorrt-cicd]

PR_Github #67566 [ run ] triggered by Bot. Commit: a3f6d8f Link to invocation

[tensorrt-cicd]

PR_Github #67566 [ run ] completed with state SUCCESS. Commit: a3f6d8f
/LLM/main/L0_MergeRequest_PR pipeline #55057 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #67597 [ run ] triggered by Bot. Commit: a3f6d8f Link to invocation

[tensorrt-cicd]

PR_Github #67597 [ run ] completed with state FAILURE. Commit: a3f6d8f
/LLM/main/L0_MergeRequest_PR pipeline #55088 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #67985 [ run ] triggered by Bot. Commit: 76abf35 Link to invocation

[tensorrt-cicd]

PR_Github #67985 [ run ] completed with state FAILURE. Commit: 76abf35
/LLM/main/L0_MergeRequest_PR pipeline #55439 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #68336 [ run ] triggered by Bot. Commit: d8a8ed1 Link to invocation