[https://nvbugs/6626655][test] Align QA test-list timeout for multimodal Kimi-K2.5 dep8 with pre-merge

[tianyuxbear]

Description

accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8] carried no TIMEOUT annotation in tests/integration/test_lists/qa/llm_function_core.txt, so it ran under the QA job-level 3600s default. The pre-merge entry for the very same test in tests/integration/test_lists/test-db/l0_dgx_b200.yml already grants it TIMEOUT (120) — a 2x mismatch between the two lists.

The case has been trending against that 60-minute ceiling on B300: the last green run took 3603.8s and the failing one 3615.6s, a ~12s margin. Accuracy is healthy throughout (80.111 versus a 77.683 threshold on the last green run), and the reported queue.get / not_empty.wait frames are just the main thread awaiting results when pytest-timeout fired, not a deadlock. So this is a wall-clock budget overrun, not a functional regression.

The test body itself is unchanged across the reported regression window, so no code fix applies here. This patch makes the two test lists agree on the budget, and drops the B200/B300 waives that were added in #17883 while the overrun was under investigation, so the case actually runs again.

Note that trimming the generation budget is not a viable alternative: MAX_NUM_TOKENS = 16384 in this class feeds SamplingParams(max_tokens=...) and is deliberately preserved via preserve_caller_max_tokens=True so the thinking-mode CoT output is not truncated by lm-eval's 512 default.

Test Coverage

No new tests. This is a test-list change only:

• TIMEOUT (120) is parsed by test_list_parser.py into pytest.mark.timeout(7200), matching the pre-merge budget.
• The two waives.txt entries for this case are removed, so it is executed again on the B200 and B300 full stages.

Dev Engineer Review

• Added TIMEOUT (120) to TestKimiK25::test_nvfp4[dep8] in tests/integration/test_lists/qa/llm_function_core.txt.
• Added a 7200-second timeout to the test implementation.
• Removed the B200 and B300 waive entries.
• The changes match the pre-merge timeout and restore test execution.
• No public API changes or unrelated scope changes are present.

QA Engineer Review

• Modified TestKimiK25::test_nvfp4[dep8] in the QA test list.
• Modified tests/integration/test_lists/waives.txt by removing the B200 and B300 entries.
• Modified the Kimi K2.5 NVFP4 accuracy test to apply a 7200-second timeout.
• The test remains covered by the QA test list.
• CBTS coverage data is unavailable.
• Verdict: needs follow-up.

[tianyuxbear]

/bot --help

[github-actions]

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label. Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Requires the ci: full pre-merge approved label on the PR (ask a member of NVIDIA/trt-llm-ci-approvers). Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline. Requires the ci: full pre-merge approved label on the PR (ask a member of NVIDIA/trt-llm-ci-approvers).

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the ci: post-merge approved PR label applied by an active member of NVIDIA/trt-llm-ci-approvers. The approval label remains in place when new commits are pushed.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label.

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e375953d-36f9-48f2-a357-a6869c610d8d

📥 Commits

Reviewing files that changed from the base of the PR and between 5b4becc and f68cb2f.

📒 Files selected for processing (1)

• tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

Walkthrough

The test metadata records a 120-second timeout, removes B200 and B300 skip entries, and sets a 7200-second timeout for the Kimi K2.5 NVFP4 accuracy test.

Changes

Kimi K25 NVFP4 test handling

Layer / File(s)	Summary
Update timeout expectation and platform waivers tests/integration/test_lists/qa/llm_function_core.txt, tests/integration/test_lists/waives.txt	The QA result includes TIMEOUT (120). The B200 and B300 skip entries are removed.
Set accuracy test timeout tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py	The Kimi K2.5 NVFP4 accuracy test has a 7200-second pytest timeout.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to f68cb

This localized test-list timeout alignment is merge-ready after normal checks and review; no actionable merge-blocking risk remains.

Suggested labels: ci: full pre-merge approved

Suggested reviewers: stanleysun639, aswinvisva, wanli-jiang

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the timeout alignment for the Kimi-K2.5 QA test and matches the primary change.
Description check	✅ Passed	The description explains the issue, solution, affected tests, timeout behavior, and test coverage; the omitted checklist does not prevent review.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[xinhe-nv]

/bot run --only-qa-verify test accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8]

[tensorrt-cicd]

PR_Github #67256 [ run ] triggered by Bot. Commit: 084ca98 Link to invocation

[tianyuxbear]

/bot run --only-qa-verify test accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8]

[tensorrt-cicd]

PR_Github #67256 [ run ] completed with state FAILURE. Commit: 084ca98
LLM_FUNCTION_AUTO_V2C #380 completed with status: 'UNSTABLE'
QA verify test: accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8] (NVBug 6626655, branch fix/6626655, fork tianyuxbear, dry_run_close=true)

Link to invocation

[tensorrt-cicd]

PR_Github #67289 [ run ] triggered by Bot. Commit: f68cb2f Link to invocation

[tensorrt-cicd]

PR_Github #67289 [ run ] completed with state FAILURE. Commit: f68cb2f
LLM_FUNCTION_AUTO_V2C #381 completed with status: 'UNSTABLE'
QA verify test: accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8] (NVBug 6626655, branch fix/6626655, fork tianyuxbear, dry_run_close=true)

Link to invocation

[xinhe-nv]

/bot run --only-qa-verify test accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8]

[tensorrt-cicd]

PR_Github #67319 [ run ] triggered by Bot. Commit: f68cb2f Link to invocation

[tensorrt-cicd]

PR_Github #67319 [ run ] completed with state SUCCESS. Commit: f68cb2f
LLM_FUNCTION_AUTO_V2C #382 completed with status: 'SUCCESS'
QA verify test: accuracy/test_llm_api_pytorch_multimodal.py::TestKimiK25::test_nvfp4[dep8] (NVBug 6626655, branch fix/6626655, fork tianyuxbear, dry_run_close=true)

Link to invocation