[AMD][AgentX] MINIMAX-M3 FP4 MI355X agentX vLLM

[ajith-sirra-amd]

Summary

• Add MiniMax-M3 FP4 single-node agentic benchmark support on MI355X using vLLM
• New script: benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.sh with lmcache (LMCache MP server + LMCacheMPConnector)
• New master config entry minimaxm3-fp4-mi355x-vllm-agentic with TP4 search space
• Image: vllm/vllm-openai-rocm:nightly-69715823df89b11ee684b84066390cbb9092d5c1
• Model: amd/MiniMax-M3-MXFP4

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🔴 perf-changelog.yaml:4607 — perf-changelog.yaml is missing its trailing newline after pr-link: To-Be-Added (the diff footer shows \ No newline at end of file). utils/validate_perf_changelog.py explicitly rejects this: parse_changelog (line 89) and validate_matrix_compatible_change (line 352-354) both raise ChangelogValidationError("{path} does not end with a newline") — so the changelog-validation job will fail and block merge. Also worth updating pr-link: To-Be-Added to the actual PR URL before merge; every other entry uses a real link.

  Extended reasoning...

  What the bug is

  The new entry appended to perf-changelog.yaml terminates without an LF byte. Byte-level inspection (od -c) confirms the file ends with the raw bytes T o - B e - A d d e d — no trailing \n. The diff itself makes this explicit with the \ No newline at end of file marker after pr-link: To-Be-Added.

  Why existing code doesn't tolerate it

  utils/validate_perf_changelog.py has two independent hard checks for this exact condition:

  • parse_changelog (line 88-90): if not raw.endswith(b"\n"): raise ChangelogValidationError(f"{label} does not end with a newline")
  • validate_matrix_compatible_change (line 350-354) re-runs the same check on the head-ref bytes: if not head_raw.endswith(b"\n"): raise ChangelogValidationError(f"{path} at {head_ref} does not end with a newline")

  AGENTS.md line 176 explicitly warns about this file: "Never delete or modify whitespace in perf-changelog.yaml — CI depends on exact whitespace (including trailing spaces on blank separator lines). Altering it breaks CI."

  Impact

  The changelog-validation job will refuse this PR — this is a merge-blocking CI failure, not a cosmetic issue. It's also the failure mode AGENTS.md explicitly warns about: a future append to this file (via >> or a non-POSIX editor that doesn't insert an LF before writing) would physically concatenate the next - config-keys: onto the same line as To-Be-Added, silently corrupting the YAML.

  Step-by-step proof

  1. git show HEAD -- perf-changelog.yaml | tail -3 shows: pr-link: To-Be-Added followed by \ No newline at end of file.
  2. tail -c 30 perf-changelog.yaml | od -c returns ...9 2 d 5 c 1 " \n p r - l i n k : T o - B e - A d d e d — the final byte is d, not \n.
  3. Running validation: python utils/validate_perf_changelog.py (or its equivalent CI entrypoint) enters parse_changelog, which reads the file as bytes and immediately hits if not raw.endswith(b"\n") on line 89 → raises ChangelogValidationError: perf-changelog.yaml does not end with a newline.
  4. Every other entry in the file terminates with \n\n before the next - config-keys: — this is the only entry that breaks the invariant.

  Fix

  Append a single \n byte to the end of the file (e.g., printf '\n' >> perf-changelog.yaml or ensure the editor's "insert final newline" setting is on). Also update pr-link: To-Be-Added to https://github.com/SemiAnalysisAI/InferenceX/pull/2129 to match every other entry's convention before merge.

[claude]

🟡 The LMCache install at minimaxm3_fp4_mi355x.sh:93 clones https://github.com/LMCache/LMCache.git with no pinned tag/SHA (--branch/--depth), so each benchmark run installs whatever HEAD of the LMCache default branch happens to be at that moment — silently invalidating cross-run perf comparability, which is the whole point of this recipe. The clone is also non-idempotent under set -euo pipefail: a retried run in the same workspace dies at git clone with 'destination path already exists'. Prefer a versioned pip install (as in kimik2.5_fp4_b200.sh:99: agentic_pip_install --quiet --no-cache-dir lmcache), or pin the clone (git clone --depth 1 --branch <tag> ...) with an rm -rf LMCache guard first (as in minimaxm3_fp8_mi325x.sh:87 for mooncake).

Extended reasoning...

What the bug is

At benchmarks/single_node/agentic/minimaxm3_fp4_mi355x.sh:91-97, the lmcache branch of the offload-backend case builds LMCache from source like this:

lmcache)
    unset VLLM_USE_SIMPLE_KV_OFFLOAD
    git clone https://github.com/LMCache/LMCache.git
    cd LMCache
    pip install -r requirements/build.txt
    CXX=hipcc BUILD_WITH_HIP=1 pip install -e .   --no-build-isolation
    cd ..

There is no --branch <tag>, no --depth 1, no post-clone git checkout <sha> — the clone resolves to whatever HEAD of the LMCache default branch (main) is at run time.

Impact — reproducibility

The entire purpose of a recipe in this repo is to produce numerically comparable perf points that get ingested into perf-changelog.yaml and rolled up across time. The single search-space row in configs/amd-master.yaml for this recipe is:

- { tp: 4, kv-offloading: dram, kv-offload-backend: lmcache, conc-list: [1, 4, 8, 16, 32] }

i.e. every configuration this recipe measures goes through the lmcache branch and therefore through the LMCache library. LMCache is an active project — LMCache/main moves. Two runs of this recipe on different days will install different LMCache revisions and can produce different tokens/s, TTFT and cache-hit numbers without any change to this repository. That is the exact anti-pattern the codebase avoids everywhere else: configs/amd-master.yaml pins every vLLM/SGLang image to a digest-suffixed nightly tag (see the dsv4-fp4-mi355x-vllm block comment that explicitly warns "pin to a digest-suffixed nightly tag rather than the floating :nightly"), and the sibling Mooncake source builds in minimaxm3_fp8_mi325x.sh:87 and minimaxm3_fp8_mi300x.sh:87 use git clone --depth 1 --branch "$mooncake_tag" ... with mooncake_tag='v0.3.11.post1'.

Impact — idempotency

The script begins with set -euo pipefail at line 2. git clone <url> into an existing non-empty directory fails with fatal: destination path 'LMCache' already exists and is not an empty directory, exit code 128. There is no rm -rf LMCache guard or [[ -d LMCache ]] || ... check. If a prior run in the same workspace didn't clean up (retried run, workspace reuse across configs in the sweep, or actions/checkout not pruning untracked directories), the second attempt aborts before the LMCache server ever starts.

Regression from the pre-existing pattern

The sibling NVIDIA recipe benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:99 uses a versioned PyPI install through the isolated agentic venv:

agentic_pip_install --quiet --no-cache-dir lmcache

This was the established pattern for installing LMCache in an agentic recipe. Switching to an unpinned HEAD clone is a strict regression. (The MI355X build does need a HIP source build — no ROCm wheel on PyPI — so falling back to a git clone is fine, but it must be pinned.)

Step-by-step proof of the non-idempotency failure

1. Sweep dispatches the config { tp: 4, kv-offloading: dram, kv-offload-backend: lmcache, conc: 1 }; the script cds into the workspace, runs git clone https://github.com/LMCache/LMCache.git, builds, benchmarks, exits 0. LMCache/ remains in the workspace.
2. Sweep dispatches the next config { tp: 4, ..., conc: 4 }; the script cds into the same workspace again, hits git clone at line 93.
3. Git prints fatal: destination path 'LMCache' already exists and is not an empty directory and returns exit 128.
4. set -e immediately aborts the script with exit 128 — wait_for_lmcache_ready, vllm serve, and run_agentic_replay_and_write_outputs never run. The config is recorded as failed.

(Whether the workspace persists across configs depends on the launcher; the current launch_mi355x-amds.sh mounts $GITHUB_WORKSPACE at /workspace, so in practice fresh CI containers usually paper over this — but any local re-run or workspace-reuse path trips the failure.)

Fix

A one-line change, either

agentic_pip_install --quiet --no-cache-dir lmcache==<version>

or, keeping the HIP source build:

rm -rf LMCache
git clone --depth 1 --branch <tag-or-sha> https://github.com/LMCache/LMCache.git

Severity

Nit — a clean container run in CI passes, the numerical result is some valid measurement, and there is no user-visible crash on the green path. But this is directly identified as an anti-pattern relative to the sibling recipes in the same directory and it silently erodes benchmark comparability over time, so worth calling out before merge.

[claude]

🟡 The docstring, check_env_vars call, and variable defaults in the new script are internally inconsistent: (1) the header lists OFFLOADING but check_env_vars actually requires KV_OFFLOADING; (2) the header omits DURATION, EP_SIZE, DP_ATTENTION, and KV_OFFLOAD_BACKEND, all of which check_env_vars enforces; (3) the DURATION=${DURATION:-1800} and EP_SIZE=${EP_SIZE:-1} defaults at lines 19-20 are dead code (check_env_vars has already exited if either was empty); and (4) the header enumerates none / cpu / lmcache as OFFLOADING values but the case statement switches on $KV_OFFLOAD_BACKEND and only implements the lmcache arm — cpu and none silently fall through to --no-enable-prefix-caching. The launcher passes the correct vars so CI is unaffected, but the header will actively mislead anyone reading or running this stand-alone. Fix by syncing the docstring with the actual required vars and either dropping the unreachable defaults or removing DURATION/EP_SIZE from check_env_vars.

Extended reasoning...

The script header at lines 5-20 sets up a runtime interface that contradicts itself on four related points. This is doc/dead-code inconsistency in a brand-new file, so nothing breaks in the CI-driven flow — but this recipe is very likely to be copy-pasted (it is the first vLLM+LMCache MI355X agentic entry), and future readers will be misled.

(1) Wrong variable name in the required-vars list. Line 7 documents OFFLOADING, but line 14 calls check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION. Sibling recipes (dsv4_fp4_b300_sglang.sh:7, kimik2.5_fp4_b300.sh:10, qwen3.5_fp8_h100.sh:15, minimaxm3_fp8_mi325x.sh) all consistently use the KV_OFFLOADING name — the header here is out of sync with the codebase convention.

(2) Missing required vars. The header lists 6 vars; check_env_vars enforces 9. DURATION, EP_SIZE, and DP_ATTENTION are all required at line 14 but never mentioned in the header. A user reading only the header, exporting the 6 documented vars, and running the script will hit Error: The following required environment variables are not set: DURATION EP_SIZE DP_ATTENTION.

(3) Dead-code defaults. Lines 19-20 set DURATION=${DURATION:-1800} and EP_SIZE=${EP_SIZE:-1}. But check_env_vars (benchmarks/benchmark_lib.sh:234) uses [[ -z "${!var_name:-}" ]] to detect unset OR empty and exits 1 if any listed var is missing. So by the time execution reaches line 19, DURATION and EP_SIZE are guaranteed non-empty — the :- fallbacks are unreachable. Compare minimaxm3_fp8_mi325x.sh:7, which requires the same vars in check_env_vars and correctly does NOT repeat the defaults.

(4) Values list documents a third variable name. Lines 9-12 enumerate none / cpu / lmcache as OFFLOADING values. The case statement at line 90 switches on $KV_OFFLOAD_BACKEND — a third, undocumented name — and only implements the lmcache arm. Setting KV_OFFLOAD_BACKEND=cpu (or =none) falls through to the initial OFFLOAD_ARGS=(--no-enable-prefix-caching) from line 47, which is neither vLLM native CPU offload nor GPU-KV-only in any meaningful sense.

Step-by-step proof. Take the header at face value and run: MODEL=amd/MiniMax-M3-MXFP4 TP=4 CONC=8 OFFLOADING=lmcache TOTAL_CPU_DRAM_GB=3000 RESULT_DIR=/tmp/x ./minimaxm3_fp4_mi355x.sh. set -u is on, so line 14 checks KV_OFFLOADING (which the user did not set because the header said OFFLOADING); check_env_vars reports 4 missing vars: KV_OFFLOADING, DURATION, EP_SIZE, DP_ATTENTION. Even if the user then also exports KV_OFFLOADING=lmcache DURATION=1800 EP_SIZE=1 DP_ATTENTION=false, the case at line 90 needs KV_OFFLOAD_BACKEND=lmcache — still-undocumented — otherwise LMCache never starts and vLLM runs with --no-enable-prefix-caching only.

Fix. Two clean options: (a) sync the header — replace OFFLOADING with KV_OFFLOADING, add KV_OFFLOAD_BACKEND, DURATION, EP_SIZE, DP_ATTENTION to the required list, drop the invented cpu/none values (or implement them), and delete lines 19-20; or (b) treat DURATION/EP_SIZE as optional-with-defaults — remove them from check_env_vars, keep the defaults, and add them to the header as optional. The sibling minimaxm3_fp8_mi325x.sh follows option (a); either is fine so long as the docstring and code match. Non-blocking (nit) — CI passes because configs/amd-master.yaml supplies the right variables via the launcher, so this only bites stand-alone users and future readers.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29024988092
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29024988092

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29032224772
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29032224772

[ajith-sirra-amd]

/reuse-sweep-run

[seungrokj]

@ajith-sirra-amd
#2118
looks better than this. Why don't we go with PR2118 ?