[AMD][AgentX] MINIMAX-M3 FP4 MI355X agentX vLLM#2118
Conversation
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
ajith-sirra-amd
requested review from
1am9trash,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
seungrokj
changed the title
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 | ||
|
|
||
| VLLM_CMD=( | ||
| vllm serve "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| "${PARALLEL_ARGS[@]}" | ||
| --trust-remote-code | ||
| --block-size 128 | ||
| --gpu-memory-utilization 0.85 | ||
| --language-model-only | ||
| --attention-backend TRITON_ATTN | ||
| --moe-backend aiter | ||
| --kv-cache-dtype fp8 | ||
| --tool-call-parser minimax_m3 | ||
| --enable-auto-tool-choice | ||
| --reasoning-parser minimax_m3 | ||
| --max-num-seqs "$CONC" | ||
| "${OFFLOAD_ARGS[@]}" | ||
| ) |
There was a problem hiding this comment.
🔴 The new script launches vllm serve "$MODEL" at line 243, but the block-header comment at line 40 explicitly states "MODEL_PATH is what the server is launched with" and every other vllm-based agentic recipe in this directory uses vllm serve "$MODEL_PATH" --served-model-name "$MODEL". On the target mi355x-amds cluster where launchers pre-stage the model into a non-default MODEL_PATH, this bypasses the pre-staged directory and re-resolves via the default HF cache. Fix by replacing "$MODEL" with "$MODEL_PATH" --served-model-name "$MODEL".
Extended reasoning...
Bug
benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.sh:243 launches the vLLM server with:
VLLM_CMD=(
vllm serve "$MODEL"
...
)But lines 40–48 of the same file establish MODEL_PATH as the canonical launch location:
# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
else
hf download "$MODEL"
export MODEL_PATH="$MODEL"
fiThe comment at line 40 explicitly promises "MODEL_PATH is what the server is launched with," yet the actual vllm serve invocation ignores MODEL_PATH entirely.
Divergence from all sibling recipes
Every other vllm-based agentic recipe under benchmarks/single_node/agentic/ uses the established pattern vllm serve "$MODEL_PATH" --served-model-name "$MODEL":
| File | Line |
|---|---|
kimik2.5_fp4_mi355x.sh |
94 |
minimaxm3_fp8_mi325x.sh (closest topological sibling) |
177 |
minimaxm3_fp8_mi300x.sh |
167 |
minimaxm3_fp8_h100.sh |
96 |
minimaxm3_fp8_h200.sh |
140 |
dsv4_fp4_b200_vllm.sh |
196 |
dsv4_fp4_b300_vllm.sh |
199 |
dsv4_fp8_h200.sh |
49 |
kimik2.5_fp4_b200.sh |
169 |
kimik2.5_fp4_b300.sh |
59 |
kimik2.5_int4_{b200,h100,h200}.sh |
51 / 50 / 58 |
Only minimaxm3_fp4_mi355x.sh:243 diverges. Given that the MODEL_PATH prep block (lines 41–48) is copy-identical to minimaxm3_fp8_mi325x.sh, this looks like a copy-paste that missed the corresponding vllm serve update.
Impact — step-by-step proof
Consider a run driven by the minimaxm3-fp4-mi355x-vllm-agentic entry in configs/amd-master.yaml, which pins runner: cluster:mi355x-amds. On that cluster the launcher pre-stages amd/MiniMax-M3-MXFP4 to a shared scratch location and exports something like MODEL_PATH=/it-share/models/amd/MiniMax-M3-MXFP4:
- Line 34:
hf download "$MODEL"unconditionally populates the default HF cache (~/.cache/huggingface/hub/or$HF_HUB_CACHE). - Lines 41–48: because
MODEL_PATHis set and (assume) already populated, the block is a no-op — the intent is clearly to launch from the pre-staged directory. - Line 243:
vllm serve "$MODEL"is called with the HF IDamd/MiniMax-M3-MXFP4. vLLM resolves that viasnapshot_download, hitting the default HF cache — not$MODEL_PATH. - Result: the pre-staged directory the launcher took care to populate is bypassed. Either (a) the HF cache already holds the snapshot (silent duplication, wasted disk and cluster bandwidth), or (b) the pre-staged snapshot differs (different revision, custom quantization, hand-patched config) — in which case a different model is loaded than what the recipe intends. There is no error message; the server comes up and reports success.
A secondary consequence: the OpenAI-compatible API exposes the checkpoint under its filesystem path, not $MODEL. Downstream clients written against the sibling recipes will expect amd/MiniMax-M3-MXFP4 as the served model name and get a mismatch.
Fix
One-line change at benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.sh:243:
VLLM_CMD=(
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
...
)This matches every sibling recipe, honors the file's own line-40 comment, and restores the pre-staging infrastructure for the mi355x-amds cluster this recipe targets.
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
…& Fixing MODEL_PATH Arg in Serve Command. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
…& Fixing MODEL_PATH Arg in Serve Command. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28942489112 |
…hmark Drop lmcache case branch, helper functions, and config entry; keep native-only offload path. Add OFFLOAD_ARGS fallback, enable prefix caching and hybrid KV cache manager, bump max-num-seqs to 2x CONC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28987547078 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28992184884 |
…cache flag. Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
| TOTAL_CPU_DRAM_GB="${TOTAL_CPU_DRAM_GB:-3000}" | ||
| TOTAL_CPU_DRAM_PARTITION_GB="${TOTAL_CPU_DRAM_PARTITION_GB:-${TOTAL_CPU_DRAM_GB}}" | ||
|
|
||
| OFFLOAD_ARGS=( | ||
| --kv_offloading_backend native | ||
| --kv_offloading_size "$TOTAL_CPU_DRAM_PARTITION_GB" |
There was a problem hiding this comment.
@cquil11 dont u need to use dram that is porrtional to # of gpus used?
There was a problem hiding this comment.
@functionstackx : The CPU DRAM GB calculation ( proportional to TP value ) is already accounted here : https://github.com/SemiAnalysisAI/InferenceX/blob/main/utils/matrix_logic/generate_sweep_configs.py#L114-L116
seungrokj
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28995437941 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28995437941 |
Signed-off-by: ajith-sirra-amd <ajith.sirra@amd.com>
…nalysisAI/InferenceX into amd/agentx-minimax-m3-vllm
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29016569168 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29016569168 |
cquil11
added
the
agentx
ChatGPT Codex Connector
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29031361208 |
|
/reuse-sweep-run |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29031361208 |
|
/reuse-sweep-run |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29031361208 |
4 participants
Summary
benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.shwith three KV offload backends:none,native(vLLM OffloadingConnector),lmcache(LMCache MP server + LMCacheMPConnector)minimaxm3-fp4-mi355x-vllm-agenticwith TP4 search space acrossnone,native, andlmcachebackendsvllm/vllm-openai-rocm:nightly-69715823df89b11ee684b84066390cbb9092d5c1amd/MiniMax-M3-MXFP4