[AgentX][vLLM][NVIDIA] chore: tune vLLM DeepSeek v4 agentic B200+B300#2138
[AgentX][vLLM][NVIDIA] chore: tune vLLM DeepSeek v4 agentic B200+B300#2138cquil11 wants to merge 20 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Claude finished @cquil11's task in 5m 8s —— View job Review of PR #2138
SummaryOne issue found, flagged inline in both scripts (🟡): the new Everything else checks out:
Two minor cautions (no inline comment, CI will exercise them): Review limitations: the vLLM source MCP server was unavailable and |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --numa-bind | ||
| --enable-cumem-allocator | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' |
There was a problem hiding this comment.
🟡 WARNING: This replaces the previous --attention_config.use_fp4_indexer_cache=True with a full --attention-config JSON that omits use_fp4_indexer_cache, silently reverting the FP4 indexer cache to its default.
Why it matters: Every other DSV4 vLLM recipe in this repo enables it — including the GB200 disagg agentic recipes that use the exact same backend, e.g. benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-3p2d-tep8-tp8-agentic.yaml:103 sets {"backend": "FLASHINFER_MLA_SPARSE_DSV4", "use_prefill_query_quantization": true, "use_fp4_indexer_cache": true}. This script also still carries the CUTLASS DSL reinstall workaround for DSV4 indexer compilation (lines 69–73), so the indexer path is clearly still active. Dropping the FP4 indexer cache changes KV memory footprint and perf on the very sweep this PR is tuning, and the PR description doesn't mention removing it.
Fix: If the drop is intentional, say so in the PR/changelog; otherwise:
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' | |
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}' |
| --numa-bind \ | ||
| --enable-cumem-allocator \ | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' \ | ||
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' \ |
There was a problem hiding this comment.
🟡 WARNING: Same as the B200 script — the new --attention-config JSON drops the previously enabled use_fp4_indexer_cache, and the PR description doesn't mention removing it. Peer DSV4 agentic recipes with this backend set "use_fp4_indexer_cache": true (see disagg-gb200-3p2d-tep8-tp8-agentic.yaml:103).
Fix: If intentional, document it; otherwise:
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' \ | |
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}' \ |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29049472753 |
cquil11
changed the title
cquil11
changed the title
| --enable-cumem-allocator | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' | ||
| --moe-backend deep_gemm_mega_moe |
There was a problem hiding this comment.
🔴 --moe-backend deep_gemm_mega_moe is now added unconditionally to the vllm serve command in both dsv4_fp4_b200_vllm.sh (line 209) and dsv4_fp4_b300_vllm.sh (line 213), but every sibling DSV4 vLLM launcher in benchmarks/single_node/fixed_seq_len/ gates this exact flag on DP_ATTENTION=true. The expanded search space still has pure-TP legs on both B200 and B300 that will now force Mega-MoE outside the DEP recipe — which per perf-changelog PR #1304 has caused CUDA symmetric-memory rendezvous failures during engine startup. Please move the flag into an MOE_ARGS=() array gated on DP_ATTENTION=true to match the fixed_seq_len pattern.
Extended reasoning...
The bug
Both AgentX launchers now emit --moe-backend deep_gemm_mega_moe as an unconditional flag inside VLLM_CMD/vllm serve:
benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:209benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh:213
However, every sibling DSV4 vLLM launcher in the repo gates this exact flag on DP_ATTENTION=true:
benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm.sh:47-50benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm.sh:55-58benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm_mtp.sh:49-53benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_vllm_mtp.sh:49-51
The b200 MTP variant documents the rationale inline: "Mega-MoE backend and the lower GMU only kick in on the DP-attn path, per the vLLM v0.20.0 DeepSeek-V4-Pro recipe."
Why the gate is load-bearing
Two perf-changelog.yaml entries record why this flag must be conditional:
- Line 1952 (PR dsv4-fp4-b200-vllm: bump to vllm v0.20.0, deep_gemm_mega_moe MoE #1204): "Gate
--moe-backend deep_gemm_mega_moeand--gpu-memory-utilization 0.85onDP_ATTENTION=trueper the v0.20.0 recipe." - Lines 2308-2309 (PR Add DSV4 B300 Dynamo vLLM disagg #1304): "Turn off explicit
deep_gemm_mega_moebackend selection in B300 Dynamo vLLM throughput recipes — Let vLLM choose the MoE backend automatically to avoid CUDA symmetric-memory rendezvous failures during startup."
The multi-node disagg-gb300-1p6d-dep4-tp4-agentic.yaml mirrors the same pattern: the DEP prefill role sets moe-backend: deep_gemm_mega_moe, but the pure-TP decode role omits it entirely.
Impact on the expanded search space
The matrix expansion in this PR keeps several pure-TP rows (no ep, no dp-attn) that will now launch with Mega-MoE forced:
- B200:
{ tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5] }and{ tp: 8, kv-offloading: dram, kv-offload-backend: mooncake, conc-list: [8, 10, 16, 32, 40, 44] } - B300:
{ tp: 4, kv-offloading: none, conc-list: [...] },{ tp: 4, kv-offloading: dram, ... },{ tp: 8, kv-offloading: none, conc-list: [1..72] },{ tp: 8, kv-offloading: dram, conc-list: [52] }.
Roughly half of the newly-expanded concurrency points are pure-TP. On those, per PR #1304's history, vLLM engine startup can hit CUDA symmetric-memory rendezvous failures; even when it doesn't, the recipe restricts this backend to DEP layouts so throughput numbers on the TP legs would not be comparable to the intended baseline.
Proof — walk through a concrete sweep point
Take the B300 point { tp: 8, ep: (unset → 1), dp-attn: (unset → false), kv-offloading: none, conc: 40 }:
EP_SIZEdefaults to 1 →EP_ARGS=()(line 195) — no--enable-expert-parallel.DP_ATTENTIONdefaults to false →PARALLEL_ARGS=(--tensor-parallel-size 8 --data-parallel-size 1)(line 189), and the vLLM router branch is skipped (line 82).- The
vllm serveinvocation still includes--moe-backend deep_gemm_mega_moeunconditionally (line 213). - Result: an 8-way pure-TP engine is launched with the DEP-only MoE backend — exactly the configuration PR Add DSV4 B300 Dynamo vLLM disagg #1304 flagged as causing symmetric-memory rendezvous failures. In the fixed_seq_len sibling launcher (
fixed_seq_len/dsv4_fp4_b300_vllm.sh:55-58) theif [ "${DP_ATTENTION}" = "true" ]; then MOE_ARGS=(--moe-backend deep_gemm_mega_moe); figuard would have keptMOE_ARGS=()empty for this same point, letting vLLM auto-select the backend.
Suggested fix
Introduce an MOE_ARGS array matching the fixed_seq_len pattern, e.g.:
MOE_ARGS=()
if [ "${DP_ATTENTION}" = "true" ]; then
MOE_ARGS=(--moe-backend deep_gemm_mega_moe)
fiand drop the flag from VLLM_CMD / vllm serve, splicing in "${MOE_ARGS[@]}" next to "${EP_ARGS[@]}". This restores the DP-attention-only gate and preserves the intended behavior on the pure-TP legs of the sweep.
| --numa-bind | ||
| --enable-cumem-allocator | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' |
There was a problem hiding this comment.
🔴 The new JSON --attention-config on line 208 (and the matching line 212 in dsv4_fp4_b300_vllm.sh) drops the use_fp4_indexer_cache key that the prior --attention_config.use_fp4_indexer_cache=True flag set, silently disabling the FP4 indexer cache optimization on the very AgentX runs this PR is tuning. Every other DSV4 FLASHINFER_MLA_SPARSE_DSV4 recipe in the repo (multi-node agentic disagg-gb200/gb300 YAMLs, fixed_seq_len B200/B300, speedbench B300) still sets this to true, and neither the PR description nor the perf-changelog notes removing it as intentional. Add "use_fp4_indexer_cache":true to the JSON in both files.
Extended reasoning...
What the bug is. The PR migrates --attention_config.use_fp4_indexer_cache=True (dot-notation) into a single JSON --attention-config blob but only carries backend and use_prefill_query_quantization forward:
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}'
use_fp4_indexer_cache is not in the new JSON, so on vllm serve it falls back to the vLLM default (not true). This affects both benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:208 and benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh:212.
Why the previous code did not prevent it. Pre-PR both files explicitly enabled the flag via --attention_config.use_fp4_indexer_cache=True. The migration to JSON was a mechanical syntax change — it should have listed all three keys — but only two survived.
Cross-repo consistency shows the intended value is true. Every other DSV4 recipe in the tree that uses FLASHINFER_MLA_SPARSE_DSV4 explicitly sets use_fp4_indexer_cache: true:
benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb200-2p1d-dep8-dep8-agentic.yamllines 108 and 131 — the analogous multi-node AgentX recipe, same backend, sameuse_prefill_query_quantization: true, plususe_fp4_indexer_cache: true.disagg-gb200-3p2d-tep8-tp8-agentic.yamllines 103 and 122 — same pattern on prefill and decode.disagg-gb300-1p6d-dep4-tp4-agentic.yamlline 92 — same.benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_vllm.shand its B300 counterpart still pass--attention_config.use_fp4_indexer_cache True.benchmarks/single_node/speedbench/dsv4_fp4_b300_vllm.sh:243still passes it.perf-changelog.yaml:1951documents this as part of the standard DSV4 recipe;perf-changelog.yaml:1750notes it is only omitted on H200 (no FP4 path).
The PR body says the goal is to "follow the DeepSeek-V4 low-latency vLLM recipe" and match the SGLang/multi-node topology — but the multi-node analog it's supposed to mirror pairs use_prefill_query_quantization: true with use_fp4_indexer_cache: true in the same JSON. So the pattern is not "newer runs drop it."
Step-by-step proof.
git show HEAD^:benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.shcontains--attention_config.use_fp4_indexer_cache=True. The FP4 indexer cache was enabled.- In the PR diff, that line is replaced by
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}'— with nouse_fp4_indexer_cachekey. - When vLLM parses
--attention-config, unspecified keys take library defaults. Nothing in the vLLM v0.24.0 release notes indicates the default foruse_fp4_indexer_cacheflipped totrue— and if it had, all the peer recipes shipping alongside the same v0.24.0 bump would not still pass it explicitly. So the effective value on B200/B300 AgentX becomes the non-truedefault. - The perf-changelog entry (
perf-changelog.yaml:4700+) lists sparse-attention, Mega-MoE, cudagraph, cuMem/NUMA, fastsafetensors, and scheduler tuning — no mention of disabling the FP4 indexer cache.
Impact. Silent perf regression on the DSV4 sparse-attention path for the exact B200/B300 AgentX runs this PR is tuning. AgentX numbers will report a partially-applied DSV4 recipe versus the intended one that peer recipes still use.
Fix. Add "use_fp4_indexer_cache":true to both JSON blobs:
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29049472753 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29050002477 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29051718384 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29052892041 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29053559526 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055933511 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29063735958 |
This reverts commit bee86d5.
1 participant
Summary
Validation
bash -n benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.shbash -n benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.shpython -m pytest utils/matrix_logic/ -q— 188 passed中文说明
验证
bash -n语法检查python -m pytest utils/matrix_logic/ -q— 188 项测试通过