[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na… - #17031
[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na…#17031aswinvisva wants to merge 13 commits into
Conversation
|
/bot run --stage-list "DGX_B200-PyTorch-4" |
WalkthroughThe VideoMME E/PD test now runs repetitions 20 through 39 and records periodic thread stacks during execution. B200 and H100 test lists select the repetitions, QA entries use ChangesVideoMME repetition coverage
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
|
PR_Github #62622 [ run ] triggered by Bot. Commit: |
|
PR_Github #62622 [ run ] completed with state |
|
The two lines removed are Two other gaps against the description: nvbugs/6327718 is still P0 Could you attach a passing run of the fp8 variant on B200 and H20 (host + commit + pytest command), the way repair-bot did for nvfp4? For what it's worth the fp8 variant already runs unwaived pre-merge via |
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
This looks like a no-op. [tests/integration/test_lists/waives.txt:16](https://github.com/NVIDIA/TensorRT-LLM/pull/17031/files#diff-621bd2af82a3b97c7a5948368d36c14582ffdf73361deb7f42d2ff395b22167eR16) still waives the same node ID unconditionally (accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8], nvbugs/6478692), so the test stays skipped on every platform and CI can't confirm the fix. Also, the fp8 param only appears in test-db/l0_h100.yml:164 and qa/llm_function_core.txt:86 — l0_b200.yml lists only the qwen3vl/nvfp4 variants — so the B200 entry was already vestigial. Worth confirming the 6327718 signature matches what you believe is fixed, and updating that bug alongside the removal.
08015b1 to
eefdfc4
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65073 [ run ] triggered by Bot. Commit: |
|
PR_Github #65073 [ run ] completed with state
|
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65111 [ run ] triggered by Bot. Commit: |
|
PR_Github #65111 [ run ] completed with state |
eefdfc4 to
9264ea2
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
1 similar comment
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
9264ea2 to
6dc8867
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #66663 [ run ] triggered by Bot. Commit: |
|
PR_Github #66663 [ run ] completed with state
|
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #66786 [ ] completed with state |
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #66867 [ run ] triggered by Bot. Commit: |
|
/bot run --stage-list "DGX_B200-PyTorch-1, DGX_B200-PyTorch-2, DGX_B200-PyTorch-3, DGX_B200-PyTorch-4, DGX_B200-PyTorch-5, DGX_B200-PyTorch-6, DGX_B200-PyTorch-7, DGX_B200-PyTorch-8, DGX_B200-PyTorch-9" --disable-fail-fast |
|
PR_Github #66898 [ run ] triggered by Bot. Commit: |
|
PR_Github #66867 [ run ] completed with state |
|
PR_Github #66898 [ run ] completed with state
|
|
/bot run --stage-list "DGX_B200-PyTorch-1, DGX_B200-PyTorch-2, DGX_B200-PyTorch-3, DGX_B200-PyTorch-4, DGX_B200-PyTorch-5, DGX_B200-PyTorch-6, DGX_B200-PyTorch-7, DGX_B200-PyTorch-8, DGX_B200-PyTorch-9" --disable-fail-fast |
1 similar comment
|
/bot run --stage-list "DGX_B200-PyTorch-1, DGX_B200-PyTorch-2, DGX_B200-PyTorch-3, DGX_B200-PyTorch-4, DGX_B200-PyTorch-5, DGX_B200-PyTorch-6, DGX_B200-PyTorch-7, DGX_B200-PyTorch-8, DGX_B200-PyTorch-9" --disable-fail-fast |
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
…est list Waive verifier rejects [qwen3vl_2b_instruct], [nemotron_nano_v3_omni_fp8], [nemotron_nano_v3_omni_nvfp4] because the parametrize decorator now expands them into per-rep IDs only (rep160..rep259). The base names no longer match any collected test. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
…59, faulthandler.enable) Prior run (rep160-259) caught a SIGSEGV on nvfp4-rep234 but did not capture any stack because faulthandler.enable() was never called. The dump_traceback_later() call only installs a periodic timer; it does NOT install SIGSEGV/SIGABRT/SIGFPE handlers, so the crash produced no diagnostic output. This commit: - Adds faulthandler.enable(file=sys.stderr, all_threads=True) so a SIGSEGV dumps every thread stack to stderr before the process dies. - Bumps rep range 160-259 -> 260-359 to bypass CI reuse-cache. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
…s.stderr (rep360-459) The prior CI run (b88b457, rep260-359) reproduced the SIGSEGV on nvfp4-rep352 but produced no faulthandler thread dump. Root cause: passing sys.stderr to faulthandler.enable() writes the crash dump into pytest --capture=fd buffered output, which is discarded when a signal kills pytest before the buffer is flushed. Fix: open a real file (under pytest --output-dir when available, else $TMPDIR) and hand its fd to faulthandler; the signal handler writes directly to that fd before the process dies. The output-dir path is uploaded post-teardown to the test-results tar even when the shard fails, so the crash log will be recoverable. Bumps rep range 260-359 -> 360-459 to bypass CI reuse cache. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
Prior run (78a01ab, rep360-459) completed 28 videomme reps clean, 0 crashes. Statistically consistent with historical ~3% base rate (P(0 in 28) = 43%). Retry with a fresh rep range to keep sampling. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
Prior runs at 100 reps only yielded ~11 videomme reps per shard due to pytest-split with 9 groups distributing across shards, while only 2 shards (DGX_H100-PyTorch-1 and DGX_B200-PyTorch-4) run per /bot run request. 500 reps in the parametrize means ~55 videomme reps per shard = ~110 total per CI run, giving ~96% probability of catching the crash at the historical ~3% failure rate. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
Prior run against commit 1f89f35 (500-rep parametrize) completed 140 videomme reps clean, 0 SIGSEGV/failures. Empty commit forces a fresh CI run so we can re-sample and either confirm crash rate stayed low OR finally reproduce a stack. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
… capture
Layers existing in-tree debug knobs on top of the current faulthandler
setup so any rep934-style deadlock produces enough evidence to root-cause
without re-running:
- sitecustomize.py bootstrap propagates faulthandler + persistent stderr
into every MPI worker, so silent worker crashes leave a real file
(fixes --capture=fd swallowing).
- TRTLLM_WORKER_PRINT_STACKS_PERIOD=60 dumps all Python threads inside
each MPI worker every 60s.
- TLLM_LLMAPI_ZMQ_DEBUG=1 logs proxy socket send/recv.
- TLLM_TRACE_EXECUTOR_LOOP=1 traces per-step executor progress.
- TORCH_NCCL_DUMP_ON_TIMEOUT / trace-buffer / debug-info-temp-file
dumps NCCL communicator state on collective timeout.
- Best-effort py-spy install for native-stack dumps; failure is silent.
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
TLLM_LLMAPI_ZMQ_DEBUG and TLLM_TRACE_EXECUTOR_LOOP inflate the Jenkins console 10-100x per rep; across ~500 videomme reps that risks truncating the tail where a crash actually lands. The framework's built-in --periodic-hang-traceback already dumps main-thread stacks on hang, so the remaining unique value of this patch is: - worker-side faulthandler (SIGSEGV/SIGABRT stacks in MPI workers) - persistent stderr (survives --capture=fd on worker death) - TORCH_NCCL_DUMP_ON_TIMEOUT (quiet on success) None of those need the trace env vars. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
Prior instrumentation left every one of ~85 rep632-style H100 fp8 crashes
in PR_Github #66364 with a 0-byte faulthandler file: the SIGSEGV came from
inside CUDA driver code, so control never returned to Python and no
Python-registered signal handler fired.
Widen the capture surface in the per-rep sitecustomize.py written for each
Python child process (including mpi4py workers):
- RLIMIT_CORE = unlimited so the kernel can write a core file next time
- py-spy watchdog thread dumping --native every 30s (walks the process
externally, so native frames survive when Python does not)
- py-spy dump chained onto the SIGSEGV/SIGABRT/SIGBUS/SIGFPE/SIGILL
handler, before we re-raise via SIG_DFL so a core file is still written
- shorter faulthandler.dump_traceback_later interval (60s -> 30s) so we
get more samples before the crash
Nothing new to install in the CI container: py-spy is already
best-effort-installed a few lines below and SYS_PTRACE is already granted
in the L0_Test podTemplate.
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
The instrumented CI run PR_Github #66663 crashed at rep950 exactly like
PR_Github #66364 crashed at rep632, but this time the L0_MergeRequest_PR
test-results tarball came back at 1 KB — DGX_H100 stage's per-rep
hang_dumps/faulthandler/sitecustomize files never made it out of the
SLURM node. The failure_analysis mentioned lost-contact / walltime, so
the working hypothesis is that the SLURM job's filesystem was torn down
before Jenkins post-stage rsync could run.
In-process capture (sitecustomize -> faulthandler + py-spy + core file)
depends on the SLURM node surviving through Jenkins archival. Add a
setsid-detached reaper subprocess that survives even a native pytest
SIGSEGV/SIGABRT and every 5s tars {hang_dumps,sitecustomize}_rep{N}
into --output-dir/nvbug6327718_snapshot.tgz via atomic tmp+mv.
Since the snapshot lives inside --output-dir, if the SLURM node
survives even a few seconds longer than the crash it will get picked
up by CI archival. And the snapshot is at most 5s stale.
Nothing else changes about the crash-capture pipeline itself.
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
… down LLMs
launch_multimodal_encoder_pd_llm() put the request thread pool on an
ExitStack but wrapped the two LLMs in a nested `with encoder, pd_llm:`.
Being inner, that nested block always unwinds first, so the pool outlived
both LLMs:
old: pd_llm -> encoder -> thread_pool -> env
new: thread_pool -> pd_llm -> encoder -> env
When an MPI worker dies mid-run, generate_async().result() raises, the
nested `with` tore down each proxy, and GenerationExecutorProxy.shutdown()
-> ZeroMqQueue.close() ran socket.close()/context.term() while this pool's
threads were still inside proxy.submit() -> ipc.py _send_data() ->
socket.send() on those same sockets. Destroying a ZMQ socket under a live
sender trips libzmq's signaler.cpp:238 assert -> abort() (exit 134), which
the CI harness records as the opaque "Test terminated unexpectedly"
(jenkins/scripts/generate_timeout_xml.py fabricates that record whenever
pytest dies without writing a result).
Put all three on the one ExitStack, entered LLMs-first and pool-last, so
the pool is drained before either proxy shuts down. MyThreadPoolExecutor
.__exit__ cancels rather than waits on the exception path, so draining
first cannot deadlock on futures blocked against an already-dead engine.
This removes the sender side of the race from the test. The receiver side
- the dispatcher thread that proxy.py:787 deliberately leaks after a
bounded join, still parked in ipc.py _recv_data() - is internal to the
proxy and unaffected; that presents as context.term() hanging rather than
aborting. Whether that remains a problem in practice is what the
repetition sweep on this branch is measuring.
Test-only change; no product code touched.
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
Measurement-only commit on the stress branch. Arm 1 (build 54455) ran the teardown-ordering fix alone at the default 128 workers over 9 B200 shards and measured 1 `Test terminated unexpectedly` in 482 reps (0.21%) against a 1.35% production baseline (13 / 965 on other PRs, last 14 days). The single residual was a hang, not an abort - the signature predicted for the dispatcher thread that GenerationExecutorProxy.shutdown() leaks after its bounded join and that the test cannot reach. This arm adds the same max_workers=16 that nano_omni_fp8 carries to measure whether capping in-flight work also reduces that residual. Everything else is identical to arm 1, so the two runs differ only in the worker count. Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
2cb3e60 to
5f5b31a
Compare
|
/bot run --stage-list "DGX_B200-PyTorch-1, DGX_B200-PyTorch-2, DGX_B200-PyTorch-3, DGX_B200-PyTorch-4, DGX_B200-PyTorch-5, DGX_B200-PyTorch-6, DGX_B200-PyTorch-7, DGX_B200-PyTorch-8, DGX_B200-PyTorch-9" --disable-fail-fast |
|
PR_Github #67525 [ run ] triggered by Bot. Commit: |
|
PR_Github #67525 [ run ] completed with state
|
5 participants
…no_v3_omni_fp8] on B200 and H20
The root cause of the pre-fix failure —
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a unionattensorrt_llm/executor/proxy.pywhen the test infrastructure's session-reuse cache monkey-patchesMpiPoolSessionin the proxy module namespace with a factory function — is already fixed on main. The proxy code was refactored to identify pool-backed sessions by excluding(MpiCommSession, RemoteMpiCommSessionClient)instead ofisinstance(x, MpiPoolSession).Remove the two waivers linked to NVBug 6327718 so CI can verify the fix on both platforms.
Dev Engineer Review
nvbug: 6327718waivers for the targeted test on B200 and H20.rep20throughrep39and periodic thread-stack dumps.QA Engineer Review
tests/integration/test_lists/qa/llm_function_core.txt,tests/integration/test_lists/test-db/l0_b200.yml, andtests/integration/test_lists/test-db/l0_h100.yml.test_disaggregated_videommeto accept the repetition parameter.rep20throughrep39coverage in the B200 and H100 test lists.test-db/for CI andqa/for manual QA.Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.