[https://nvbugs/6649384][test] A/B revert #16687 for B200 disaggregated stress regression - #18327
Draft
chienchunhung wants to merge 1 commit into
Draft
Conversation
…errors and fail fast on desync (NVIDIA#16687)" This reverts commit 09b77e8. Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Collaborator
Author
|
/bot run --disable-fail-fast --gpu-type "B200" |
Collaborator
|
PR_Github #69828 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #69828 [ run ] completed with state |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Investigation-only draft for NVBUG 6649384. Do not merge.
This PR tests whether #16687 is causally required for the B200 disaggregated stress regression. It applies a signed semantic revert of #16687 on the exact failing base
0f2c3a95f9415045bdf06a7230759475692483b6.The branch is intentionally kept on that historical base rather than rebased onto current
main, so the treatment remains comparable with the recorded control failure. The production control at0f2c3a95failed with 54,038/54,038 non-cancellation requests returningLLM is shutting down.0f2c3a95f9415045bdf06a7230759475692483b63245fc3ecd76e2fb610f42f2422102e2430c28feartifactory.nvidia.com/sw-tensorrt-llm-docker-local/tensorrt-llm:pytorch-26.05-py3-x86_64-ubuntu24.04-skip-tritondevel-202607311529-16970Interpretation: a treatment pass while the matched control fails strongly implicates #16687; the same failure on both variants exonerates it as the initiating cause. A changed failure mode would indicate that #16687 affects propagation/teardown but does not prove it creates the first worker error.
Test Coverage
Targeted CI test:
disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-gpt_oss_120b_eagle_trtllm_stress]PR-CI limitation: this case is registered only in
qa/llm_function_stress.txt, which is not part of the public PR-stage map. The B200-filtered PR pipeline validates that this historical revert branch builds and passes mapped B200 coverage, but the decisive A/B result still requires the targeted QA cluster job using the pinned image above.Local validation completed:
git diff --checkPR Checklist