Skip to content

[https://nvbugs/6619882][test] Unwaive unittest/_torch/sampler on DGX_B200 - #17946

Open
zhaoyangwang-nvidia wants to merge 1 commit into
NVIDIA:mainfrom
zhaoyangwang-nvidia:unwaive-sampler-b200-6619882
Open

[https://nvbugs/6619882][test] Unwaive unittest/_torch/sampler on DGX_B200#17946
zhaoyangwang-nvidia wants to merge 1 commit into
NVIDIA:mainfrom
zhaoyangwang-nvidia:unwaive-sampler-b200-6619882

Conversation

@zhaoyangwang-nvidia

@zhaoyangwang-nvidia zhaoyangwang-nvidia commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • Removed the _torch/sampler -k "not test_speculative_d2h_parity_real_predictor" waiver from tests/integration/test_lists/waives.txt.
  • The change restores DGX_B200 sampler coverage without changing test code or public entities.
  • The waiver format and scope are consistent with the stated objective.
  • A local 8xB200 run passed 1,147 tests, skipped 699, and deselected 2.

QA Engineer Review

  • Modified tests/integration/test_lists/waives.txt.
  • Removed the _torch/sampler waiver entry.
  • CBTS coverage data is unavailable.

Verdict: needs follow-up

Description

unittest/_torch/sampler -k "not test_speculative_d2h_parity_real_predictor" was waived on DGX_B200 by #17782 (a098858cef) because of an intermittent failure that shows up roughly 4 times per month.

The reported failure is not an assertion failure. In the CI log the inner pytest process dies without printing a summary or a traceback, right after the KV cache allocation of a test_beam_search_e2e[multi_process-...] case; the outer wrapper then only surfaces the generic AssertionError: failure reported in unittests.

Waiving the whole entry drops coverage for the entire sampler directory on Blackwell, which is far broader than the intermittent failure itself. This PR removes the waive so the case runs in CI again and the intermittent failure can be characterized on current main.

Test Coverage

unittest/_torch/sampler -k "not test_speculative_d2h_parity_real_predictor" (DGX_B200-PyTorch-1) is re-enabled; no other test is affected.

Local run on an 8xB200 node at bd90276f6b, bypassing the waive list:

1147 passed, 699 skipped, 2 deselected in 995.93s (0:16:35)

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…_B200

The whole `unittest/_torch/sampler` entry was waived on DGX_B200 by NVIDIA#17782
because of an intermittent failure (about 4 occurrences per month). The
reported failure is not an assertion failure: the inner pytest process dies
without emitting a summary or traceback, and the outer wrapper only surfaces
`AssertionError: failure reported in unittests`.

Waiving the whole entry removes coverage for the entire sampler directory on
Blackwell, which is much broader than the intermittent failure itself.
Unwaive it so the case runs in CI again and the intermittent failure can be
characterized on current main.

Signed-off-by: ZhaoyangWang <zhaoyangw@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8d13e97e-977c-4a48-8693-1244f6945408

📥 Commits

Reviewing files that changed from the base of the PR and between 18b0eaf and 33a6a44.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The integration waiver list removes the skip entry for test_speculative_d2h_parity_real_predictor in the _torch/sampler test group.

Changes

Sampler waiver update

Layer / File(s) Summary
Remove sampler test skip
tests/integration/test_lists/waives.txt
Removes the waiver that excluded test_speculative_d2h_parity_real_predictor from the _torch/sampler test group.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 33a6a

This change removes a test waiver and restores sampler test coverage on DGX_B200 without changing product behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

Suggested labels: ci: full pre-merge approved

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the NVBugs issue, test change, and affected sampler test group.
Description check ✅ Passed The description explains the intermittent failure, the waiver removal, the restored coverage, and the relevant test results.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #67299 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #67300 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #67299 [ run ] completed with state ABORTED. Commit: 33a6a44

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #67300 [ run ] completed with state SUCCESS. Commit: 33a6a44
/LLM/main/L0_MergeRequest_PR pipeline #54821 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #67374 [ run ] triggered by Bot. Commit: 33a6a44 Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants