Skip to content

[https://nvbugs/6649384][test] A/B revert #16687 for B200 disaggregated stress regression - #18327

Draft
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6649384-ab-revert-16687
Draft

[https://nvbugs/6649384][test] A/B revert #16687 for B200 disaggregated stress regression#18327
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:codex/nvbug-6649384-ab-revert-16687

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Description

Investigation-only draft for NVBUG 6649384. Do not merge.

This PR tests whether #16687 is causally required for the B200 disaggregated stress regression. It applies a signed semantic revert of #16687 on the exact failing base 0f2c3a95f9415045bdf06a7230759475692483b6.

The branch is intentionally kept on that historical base rather than rebased onto current main, so the treatment remains comparable with the recorded control failure. The production control at 0f2c3a95 failed with 54,038/54,038 non-cancellation requests returning LLM is shutting down.

Interpretation: a treatment pass while the matched control fails strongly implicates #16687; the same failure on both variants exonerates it as the initiating cause. A changed failure mode would indicate that #16687 affects propagation/teardown but does not prove it creates the first worker error.

Test Coverage

Targeted CI test:

disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-gpt_oss_120b_eagle_trtllm_stress]

PR-CI limitation: this case is registered only in qa/llm_function_stress.txt, which is not part of the public PR-stage map. The B200-filtered PR pipeline validates that this historical revert branch builds and passes mapped B200 coverage, but the decisive A/B result still requires the targeted QA cluster job using the pinned image above.

Local validation completed:

  • git diff --check
  • Python syntax compilation for the changed production and coupled test files

PR Checklist

  • Investigation scope, base SHA, treatment SHA, environment, target test, and interpretation criteria are documented.

…errors and fail fast on desync (NVIDIA#16687)"

This reverts commit 09b77e8.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --gpu-type "B200"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #69828 [ run ] triggered by Bot. Commit: 3245fc3 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #69828 [ run ] completed with state SUCCESS. Commit: 3245fc3
/LLM/main/L0_MergeRequest_PR pipeline #57119 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants