[https://nvbugs/6418815][test] Pin fp32-matmul precision in VisualGen LPIPS tests and re-baseline Cosmos3 goldens - #17780
Conversation
…ergence The Cosmos3 LPIPS goldens pass in CI but reproduce on no developer machine we can build. edge_t2i measures 0.117127 locally against the 0.0056 recorded at creation, while every other VisualGen golden (qwenimage at 50 steps, wan, flux) reproduces locally without trouble. Ruled out locally, each by measurement and nearly all bit-identical across the change: GPU generation (B200 vs B300), torch build (PyPI 2.11.0, 2.12.0, and NVIDIA-patched 2.13.0a0 inside an NGC container), the container itself, the checkpoint, the models root, negative and system prompts, tokenization, per-step transformer parity against diffusers, VAE round-trip, the flow schedule, the SDPA backend, diffusers/transformers versions, and test order. This instruments test_cosmos3_edge_t2i_lpips_against_golden to fingerprint the inputs and intermediates CI actually uses -- environment, checkpoint config hashes, golden bytes, conditioning text, the first and last transformer latent, and the measured score -- so they can be diffed against the local values. It reports through the assertion message and fails unconditionally because the CI report API returns empty stdout for tests that pass. To be reverted once the divergence is understood. Signed-off-by: Igor Shovkun <igshov@gmail.com>
|
/bot run --disable-fail-fast |
|
PR_Github #66551 [ run ] triggered by Bot. Commit: |
|
PR_Github #66551 [ run ] completed with state
|
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-PyTorch-Post-Merge-1" |
|
PR_Github #66554 [ run ] triggered by Bot. Commit: |
|
PR_Github #66554 [ run ] completed with state
|
The Cosmos3 LPIPS golden tests sit in an l0_b200.yml block whose terms are `stage: post_merge`, so a PR pipeline never schedules them. Two bot runs confirmed this: neither pipeline #54182 nor #54185 produced any record for test_visual_gen_cosmos3.py, including with --extra-stage "DGX_B200-PyTorch-Post-Merge-1", because the gate is in the test-db terms rather than in the stage selection. List the instrumented test in the pre_merge pytorch b200 block so the diagnostic actually runs on a PR pipeline. Remove together with the diagnostic itself. Signed-off-by: Igor Shovkun <igshov@gmail.com>
|
/bot run --disable-fail-fast |
|
PR_Github #66651 [ run ] triggered by Bot. Commit: |
|
PR_Github #66651 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #66804 [ ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #66812 [ run ] triggered by Bot. Commit: |
…er too The payload hashed only checkpoint config JSONs, so a CI checkpoint with identical configs but different weight bytes would evade it -- and weights do not reach latent[first] either, so nothing else in the payload covers them. Add per-safetensors size plus head/tail-1MiB digests and the tokenizer.json hash, closing the different-checkpoint and different-tokenizer branches. Signed-off-by: Igor Shovkun <igshov@gmail.com>
|
/bot run --disable-fail-fast |
|
PR_Github #66834 [ run ] triggered by Bot. Commit: |
|
PR_Github #66834 [ run ] completed with state
|
… LPIPS paths Root cause of the Cosmos3 LPIPS golden divergence (nvbugs 6418815, 6437341): NGC PyTorch containers default matmul TF32 on (float32_matmul_precision "high") while PyPI torch defaults it off ("highest"), and Cosmos3 is the only VisualGen model with fp32 GEMMs inside its denoising loop -- the RoPE frequency matmul (transformer_cosmos3.py:863) and the fp32 timestep embedder (:1604), both deliberately upcast for accuracy. A CI-side fingerprint proved the inputs bit-identical (same conditioning text, checkpoint, golden bytes, and initial latent) with the trajectory diverging, and a single-flag A/B moved LPIPS-to-golden 0.132 -> 0.054. bf16 compute is unaffected by the knob. Pin "highest" (measured bit-stable across torch 2.11/2.12 and B200/B300) and cuDNN TF32 in _lpips_deterministic_algorithms and around the three Cosmos3 generation helpers that do not use it, so goldens are portable across hosts instead of encoding the golden-cutting container's arithmetic. Production code inherits environment defaults unchanged; this pins tests only. This also reverts the temporary CI diagnostic instrumentation from test_cosmos3_edge_t2i_lpips_against_golden. Signed-off-by: Igor Shovkun <igshov@gmail.com>
…r pinned arithmetic Re-cut all seven runnable Cosmos3 goldens with fp32-matmul precision pinned to "highest" (see the previous commit for the root cause: NGC-vs-PyPI TF32 defaults acting on Cosmos3's fp32 GEMM islands). Cut natively on B300 at torch 2.12.0+cu130; under the pin the trajectory measured bit-stable across torch versions and GPU generations, so these goldens are host-portable rather than bound to the golden-cutting container. - nano t2i / t2v: the two stale goldens (cut 2026-06-29, invalidated by the 07-02 negative-prompt default change) -- re-cut and unwaived (nvbugs 6418815, 6437341). - nano v2v, fp8-blockwise, edge t2i/t2v/i2v: previously CI-container-bound; re-cut under the pin. Edge goldens become TRT-LLM self-goldens; cross-stack correctness stays covered by TestDiffusersParity. - Provenance JSONs rewritten with real commit/torch/flags, explicit negative_prompt, and media sha256; first provenance record for v2v. - The other 35 zip members are byte-identical (CRC-verified on repack). - l0_b200.yml gains TEMPORARY pre_merge listings (marked REMOVE BEFORE MERGE) so the post_merge-gated tests run on this PR's pipeline for verification. Out of scope: i2v_4step (checkpoint absent in CI) and nvfp4 (nvbugs/6572800, cross-model break) -- both need a pinned-flags re-cut when unblocked. Signed-off-by: Igor Shovkun <igshov@gmail.com>
|
/bot run --disable-fail-fast |
|
PR_Github #66885 [ run ] triggered by Bot. Commit: |
|
PR_Github #66885 [ run ] completed with state
|
…temporary test listings All seven re-baselined goldens self-verified at LPIPS 0.000000 (bit-exact regeneration under the pinned arithmetic; 7 passed in 19m12s on B300 native, torch 2.12.0+cu130). Record that in the provenance JSONs and remove the temporary pre_merge listings -- the tests return to their permanent post_merge home in l0_b200.yml, which is now identical to main. Signed-off-by: Igor Shovkun <igshov@gmail.com>
ishovkun
changed the title
|
PR_Github #67871 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #67886 [ run ] triggered by Bot. Commit: |
|
PR_Github #67871 [ run ] completed with state |
|
PR_Github #67886 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #67988 [ run ] triggered by Bot. Commit: |
|
PR_Github #67988 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #68023 [ run ] triggered by Bot. Commit: |
|
PR_Github #68023 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
1 similar comment
|
/bot run --disable-fail-fast |
|
PR_Github #68150 [ run ] triggered by Bot. Commit: |
|
PR_Github #68151 [ run ] triggered by Bot. Commit: |
|
PR_Github #68150 [ run ] completed with state |
|
/bot run --disable-fail-fast |
|
PR_Github #68175 [ run ] triggered by Bot. Commit: |
|
PR_Github #68151 [ run ] completed with state |
|
PR_Github #68175 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #68252 [ run ] triggered by Bot. Commit: |
|
PR_Github #68252 [ run ] completed with state |
|
✅ LFS objects already in storage (1 file) — no sync needed. These LFS-tracked files are already present in this repository's LFS storage:
|
…eshold The static-FP8 T2I golden generated under the host's torch defaults. Cosmos3 runs fp32 GEMMs inside the denoising loop (RoPE frequencies, the timestep embedder, the fp32 autocast block), so their arithmetic follows float32_matmul_precision -- "high" (TF32) in NGC containers, "highest" on PyPI torch. NVIDIA#17780 measured that single flag moving LPIPS-to-golden 0.132 -> 0.054 on the sibling Cosmos3 goldens, and applies the pin per generation path rather than from the shared determinism helper, so this path did not inherit it. The threshold moves 0.05 -> 0.1. The golden is cut on B300 (sm_103) but scheduled only on B200 (sm_100), and a stored quantized golden does not carry across that boundary: kernel selection, split-k and reduction order, and autotuner choices all change with the architecture, none of which the fp32 pin addresses. The old 0.05 was calibrated on same-host self-regeneration distance (two renders were bit-identical, LPIPS 0.000000), which measures reproducibility on the cutting host rather than portability to the running one. 0.1 gates drastic divergence; the topology assertion and the unit suites remain the feature's real protection. Signed-off-by: Igor Shovkun <igshov@gmail.com>
6 participants
Dev Engineer Review
"highest"and disables cuDNN TF32 for Cosmos3 LPIPS generation paths.i2v_4stepandnvfp4out of scope.QA Engineer Review
i2v_4step.tests/integration/test_lists/waives.txt.0.000000on B300.Description
The Cosmos3 LPIPS goldens passed in CI but reproduced on no developer machine
(0.117–0.78 vs thresholds 0.05–0.13), and the nano t2i/t2v goldens were
additionally stale since the 2026-07-02 negative-prompt default change
(waived as nvbugs 6418815 / 6437341).
Root cause: NGC PyTorch containers default matmul TF32 on
(
float32_matmul_precision="high"), PyPI torch defaults it off(
"highest"), and Cosmos3 is the only VisualGen model with fp32 GEMMs insideits denoising loop — the RoPE frequency product, the fp32 timestep embedder,
and an explicit
autocast(dtype=torch.float32)block, all deliberatelyupcast for accuracy. The goldens therefore encoded the golden-cutting
container's arithmetic. A CI-side fingerprint proved the inputs bit-identical
between CI and local with the trajectory diverging, and a single-flag A/B
moved LPIPS-to-golden 0.132 → 0.054.
Fix: pin
float32_matmul_precision("highest")andcudnn.allow_tf32onCosmos3's generation paths (tests only — production inherits environment
defaults), and re-baseline the seven runnable Cosmos3 goldens under the
pinned arithmetic, which measured bit-stable across torch 2.11/2.12 and
B200/B300. Edge goldens become TRT-LLM self-goldens (cross-stack correctness
remains covered by
TestDiffusersParity). Provenance JSONs record the realcommit/torch/flags, explicit
negative_prompt, and media sha256. The twowaivers are removed. The other 35 zip members are byte-identical
(CRC-verified on repack).
Scope of the pin. It is applied per Cosmos3 generation path rather than
inside
_lpips_deterministic_algorithms, because that helper wrapsgeneration (not just scoring) for LTX-2, HunyuanVideo, FLUX, QwenImage and
WAN as well. Pinning there would have changed generation for five model
families whose goldens are not re-cut here — and undetectably so outside an
NGC container, since
"highest"is already the PyPI default and those goldentests are post_merge-gated. The helper is byte-identical to main; blast radius
equals the goldens this PR re-cuts. (This leaves the NVFP4 feature path
unpinned, as it generates in a spawned process via the shared helper — correct
for now, since that golden is waived under nvbugs/6572800 and not re-cut.)
On the
deterministic_algorithms: true → falseJSON edits. These arecorrections, not a loosening:
_run_cosmos3_lpips_pipelinenever called_lpips_deterministic_algorithmsfor generation — only scoring goes throughit — so the previous
truerecorded a setting that was not in effect whenthose goldens were cut.
Out of scope:
i2v_4step(checkpoint absent in CI) andnvfp4(nvbugs/6572800) — both need a pinned-flags re-cut when unblocked.
Investigation evidence
tf32_matmul=True fp32_prec=highvsFalse/highest"highest"transformer_cosmos3.py: RoPE freq product, fp32 time-embedder, fp32 autocast blockTest Coverage
examples/visual_gen/test_visual_gen_cosmos3.pyLPIPS golden tests(nano t2i / t2v / v2v / fp8-blockwise, edge t2i / t2v / i2v), post-merge on
the B200 lane (
l0_b200.yml, placement unchanged).regenerate bit-exactly against the new goldens — LPIPS 0.000000 each,
7 passed. The same tests measured 0.117–0.78 against the old goldens on the
same host.
7 passed in 13:16 — confirming the rescope preserved the exact generation
arithmetic the goldens were cut under.
generate through
_lpips_deterministic_algorithms, which this PR leavesbyte-identical to main.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.