[AMD] MiniMax-M3 MXFP8 MI355X vLLM disagg: TP4 worker-ratio + serve sync / MiniMax-M3 MXFP8 MI355X vLLM 分离式：TP4 工作单元配比 + serve 参数对齐

[hongxiayang]

Summary

Retune the minimaxm3-fp8-mi355x-vllm-disagg config for MiniMax-M3 MXFP8 disaggregated serving on MI355X:

• Image bump to vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9.
• Serve-flag/env sync with the single-node minimaxm3-fp8-mi355x-vllm recipe (PR [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003): --moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 (in benchmarks/multi_node/amd_utils/models_vllm.yaml).
• P/D search space retuned to all-TP4 — dropped the TP8 and TP4/EP layouts (they regress tok/s/GPU on gfx950) and tune the prefill:decode worker ratio instead: 1k1k = balanced 1P-TP4/1D-TP4 (conc 1-1024) + decode-heavy 1P-TP4/2D-TP4 (conc 256-2048); 8k1k = balanced 1P-TP4/1D-TP4 (conc 1-512) + prefill-heavy 2P-TP4/1D-TP4 (conc 128-512). All layouts keep prefill+decode workers ≤ 3 for the 3-node mi355x-disagg pool.

Results

On-box MI355X per-GPU throughput vs the original 6/24 disagg config (PR #1762):

• 8k1k peak ~2085 → ~6003 tok/s/gpu (~2.9×)
• 1k1k peak ~1736 → ~2586 tok/s/gpu (~1.5×)

At 8k1k the disagg frontier is now roughly on par with the best single-node aggregated TP4 recipe at the throughput end. (Exploration note: vLLM V1 data-parallel — DP2×TP4 — was tested and is ~30-47% worse per-GPU than plain TP4 due to lock-step DP execution, so it is not used.)

Validation

• Manual test-config sweep run 29063432435: both multi-node eval jobs GREEN (GSM8K ~0.958) and 3/4 throughput jobs green. The 4th (1k1k balanced) failed mid-sweep with the server going unresponsive at conc≥32 — a transient flake (the identical config passed in run 29039061173); re-running.
• Relies on the container nofile ulimit fix already on main (both server and router containers) to avoid FD exhaustion at high concurrency / during eval.

Test plan

• Apply a sweep label and confirm the disagg sweep is green (re-run the 1k1k-balanced job if it flakes again).
• Eval scores within threshold (GSM8K).

中文说明

重新调优 MI355X 上 MiniMax-M3 MXFP8 分离式(disagg)配置 minimaxm3-fp8-mi355x-vllm-disagg：

• 镜像升级至 vllm/vllm-openai-rocm:nightly-2afa3f7e...。
• serve 参数/环境变量对齐单节点 minimaxm3-fp8-mi355x-vllm recipe（PR [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003）：--moe-backend aiter、--linear-backend emulation、--max-num-batched-tokens 32768、VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1、VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6。
• P/D 搜索空间全部改为 TP4 —— 移除 TP8 与 TP4/EP 布局（在 gfx950 上每 GPU 吞吐更差），改为调节 prefill:decode 的 worker 配比：1k1k 采用均衡 1P-TP4/1D-TP4（并发 1-1024）+ decode 侧加倍 1P-TP4/2D-TP4（并发 256-2048）；8k1k 采用均衡 1P-TP4/1D-TP4（并发 1-512）+ prefill 侧加倍 2P-TP4/1D-TP4（并发 128-512）。所有布局 prefill+decode worker 数 ≤ 3 以适配 3 节点池。

结果

MI355X 实测每 GPU 吞吐相较 6/24 初版分离式配置（PR #1762）：8k1k 峰值约 2085 → 6003 tok/s/gpu（约 2.9 倍），1k1k 峰值约 1736 → 2586 tok/s/gpu（约 1.5 倍）。8k1k 端已基本追平最优单节点聚合 TP4 recipe。（附：实测 vLLM V1 数据并行 DP2×TP4 因锁步执行，每 GPU 比纯 TP4 差约 30-47%，故不采用。）

验证

手动 test-config 扫描 run 29063432435：两个多节点 eval 任务全部通过（GSM8K ~0.958），4 个吞吐任务中 3 个通过；第 4 个（1k1k 均衡）在并发≥32 时服务端中途无响应——属偶发抖动（同一配置在 run 29039061173 中通过），正在重跑。依赖已合入 main 的容器 nofile ulimit 修复（服务端与路由器容器）以避免高并发/评估时的 FD 耗尽。

Made with Cursor

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

LGTM — config-only retune of a benchmark sweep with test-sweep validation; only a stale-comment nit was found.

Extended reasoning...

Overview

This PR retunes the minimaxm3-fp8-mi355x-vllm-disagg benchmark recipe: bumps the vLLM ROCm image, syncs per-worker serve flags/env with the validated single-node MiniMax-M3 MXFP8 recipe (--moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6), and replaces the mixed-TP search space with three all-TP4 P/D layouts tuning the prefill:decode worker ratio (1P/1D, 1P/2D, 2P/1D) that respect the 3-node mi355x-disagg pool constraint. Touches only 3 YAML files: benchmarks/multi_node/amd_utils/models_vllm.yaml, configs/amd-master.yaml, and perf-changelog.yaml.

Security risks

None. This is purely benchmark configuration data — no code paths, no auth/crypto/permissions changes, no new inputs from untrusted sources. The image bump moves to a specific SHA-pinned vLLM ROCm nightly (rather than a floating tag), which is the safer pattern already established elsewhere in this file.

Level of scrutiny

Low. This is a config-only PR in the benchmark sweep space, following established patterns in the file (e.g. the sister minimaxm3-fp4-mi355x-vllm-disagg recipe already uses the same TP4 P/D layout structure and worker-ratio approach). The author cites a validation sweep with both multi-node eval jobs green (GSM8K ~0.958) and provides per-GPU throughput numbers. All layouts respect the xP + yD <= 3 node budget for the 3-node mi355x-disagg pool. The comment block explaining the new topology in configs/amd-master.yaml is thorough and matches the actual search-space entries.

Other factors

The one finding from the bug hunt is a documentation nit (a parenthetical about 'TP8 and TP4 layouts' in models_vllm.yaml that is now stale since every worker in the sweep is TP4) — non-blocking, safe to address in a follow-up if desired. No CODEOWNER-critical code paths are touched. No prior reviewer comments to address.

[claude]

🟡 The comment on line 55 says '(the sweep mixes TP8 and TP4 layouts)', but this PR retunes minimaxm3-fp8-mi355x-vllm-disagg (the only consumer of the MiniMax-M3-MXFP8 model_vllm block) to use TP4 across every prefill and decode worker — only the prefill:decode worker ratio varies now. The placeholder-rewrite mechanism this comment justifies is still correct, but the parenthetical reason is stale; consider updating to e.g. '(the sweep uses TP4 workers only; server_vllm.sh still rewrites the placeholder from the master-config prefill/decode tp)'.

Extended reasoning...

What the bug is. The comment block preceding prefill_flags/decode_flags in the MiniMax-M3-MXFP8 entry (benchmarks/multi_node/amd_utils/models_vllm.yaml line 55) explains why --tensor-parallel-size 8 in the flag strings is a placeholder: server_vllm.sh sed-rewrites it to PREFILL_TP_SIZE/DECODE_TP_SIZE from the master-config prefill/decode tp, because '(the sweep mixes TP8 and TP4 layouts)'. After this PR, that parenthetical is no longer true.\n\nWhy the parenthetical is stale. The only sweep entry that consumes this MiniMax-M3-MXFP8 model_vllm block is minimaxm3-fp8-mi355x-vllm-disagg (the other minimaxm3-fp8-mi355x-*-disagg entry uses framework: atom-disagg and a different model, so it does not go through models_vllm.yaml). This PR retunes that sweep to three P/D layouts that are all TP4: 1P-TP4/1D-TP4, 1P-TP4/2D-TP4, and 2P-TP4/1D-TP4. Every removed layout was the TP8 or mixed-TP one; no TP8 worker remains anywhere in the sweep.\n\nImpact. Zero runtime impact — the sed-rewrite from PREFILL_TP_SIZE/DECODE_TP_SIZE continues to work correctly regardless of what mix the sweep uses. The stale justification is just documentation drift: a future reader will look for the TP8+TP4 mix the comment promises, not find one, and either doubt the placeholder mechanism or waste time reconciling it against the sweep.\n\nStep-by-step proof.\n1. Grep for consumers of the MiniMax-M3-MXFP8 model_vllm block: only minimaxm3-fp8-mi355x-vllm-disagg in configs/amd-master.yaml uses framework: vllm-disagg with model: MiniMaxAI/MiniMax-M3-MXFP8.\n2. Read that recipe after this PR: 1k1k has two search-space entries — {prefill: tp=4, decode: tp=4} and {prefill: tp=4, decode: tp=4, num-worker=2}; 8k1k has two — {prefill: tp=4, decode: tp=4} and {prefill: tp=4, num-worker=2, decode: tp=4}.\n3. So every worker in every layout is TP4 — the sweep does not mix TP8 and TP4 anymore. The rationale is now the worker ratio (xP:yD), which the master-config comment block correctly describes.\n\nHow to fix. Update the parenthetical to something like (the sweep uses TP4 workers only; tp is authoritative in the master-config) — the load-bearing part (that --tensor-parallel-size 8 is a placeholder rewritten from master-config prefill.tp/decode.tp) stays the same, only the reason line needs to catch up with the retuned sweep.\n\nSeverity: nit. Doc-only staleness introduced by this PR; not merge-blocking.