[AMD][MI355X] DSv4 SGLang agentic: v0.5.14 image, router, HiCache

[seungrokj]

Summary

• Bump dsv4-fp4-mi355x-sglang-agentic image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
• Align launcher env vars and server args with fixed-seq-len sibling (dsv4 attention backend, fp8_e4m3 KV cache, --disable-radix-cache, --cuda-graph-max-bs, DP-attention exports, --enable-two-batch-overlap)
• Add SGLang router for DP-attention configs (consistent-hashing, dp-aware, correlation-id routing)
• Add HiCache KV offloading support
• Add SGLANG_CMD array pattern with command logging and env-var dump
• Add capture_cache_metrics for pre/post-benchmark cache stats
• Sweep conc=48 across TP8 +/- DPA +/- HiCache

Test plan

• Trigger dsv4-fp4-mi355x-sglang-agentic full sweep on MI355X cluster
• Verify TP8 (no DPA, no HiCache) completes without errors
• Verify TP8+DPA (router) completes without errors
• Verify TP8+HiCache completes without errors
• Verify TP8+DPA+HiCache completes without errors

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 The launcher references "$MAX_MODEL_LEN" in --context-length on line 149, but benchmark_lib.sh unconditionally unset MAX_MODEL_LEN for any caller under benchmarks/single_node/agentic/ (and check_env_vars deliberately omits it). Combined with set -euo pipefail at the top of this launcher, bash aborts with "MAX_MODEL_LEN: unbound variable" before the SGLang server is ever started — breaking all four dsv4-fp4-mi355x-sglang-agentic sweep configurations (TP8 ± DPA ± HiCache) 100% of the time. Fix by dropping the --context-length line entirely, matching every other agentic launcher in the repo (none pass --context-length/--max-model-len).

Extended reasoning...

The bug

Line 149 of benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh now builds SGLANG_CMD with:

--context-length "$MAX_MODEL_LEN"

But MAX_MODEL_LEN is guaranteed to be unset at this point, and the script is running under set -euo pipefail (line 2), so parameter expansion under set -u aborts the script with MAX_MODEL_LEN: unbound variable before SGLANG_CMD is fully constructed. The SGLang server is never launched.

Why MAX_MODEL_LEN is unset

Line 15 sources benchmarks/benchmark_lib.sh. At module scope (benchmark_lib.sh:70-77) it deliberately unsets MAX_MODEL_LEN for any caller path matching */agentic/* (or SCENARIO_TYPE=agentic-coding, which also applies here per configs/amd-master.yaml):

# Agentic replays must use the model's native context limit. Ignore inherited
# workflow or shell overrides so neither the server nor AIPerf applies a cap.
_benchmark_caller="${BASH_SOURCE[1]:-}"
if [[ "$_benchmark_caller" == */agentic/* || ... || "${SCENARIO_TYPE:-}" == "agentic-coding" ]]; then
    unset MAX_MODEL_LEN

This launcher lives at benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh, so BASH_SOURCE[1] matches. The launcher's check_env_vars call at line 17 (MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION) intentionally omits MAX_MODEL_LEN — consistent with the library's stated policy that agentic replays use the model's native context. Nothing anywhere in the launcher re-assigns MAX_MODEL_LEN after the source. grep -n MAX_MODEL_LEN on this file returns exactly one hit: line 149.

How the sibling passed but this doesn't

The fixed-seq-len sibling (benchmarks/single_node/dsv4_fp4_mi355x_sglang.sh) also uses --context-length "$MAX_MODEL_LEN" and lists MAX_MODEL_LEN in its check_env_vars — that works because the sibling is not under agentic/, so benchmark_lib.sh never unsets it. The PR description notes the launcher was aligned with the fixed-seq-len sibling; unfortunately this one line's precondition (externally-supplied MAX_MODEL_LEN) does not survive the agentic path.

Step-by-step proof

1. Sweep runner exports the standard agentic env (MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE, DP_ATTENTION, SCENARIO_TYPE=agentic-coding) and execs this launcher.
2. Line 2: set -euo pipefail → nounset active.
3. Line 15: source .../benchmark_lib.sh. BASH_SOURCE[1] = benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh → matches */agentic/* → unset MAX_MODEL_LEN.
4. Line 17: check_env_vars succeeds (it does not list MAX_MODEL_LEN).
5. Around line 149, bash evaluates the SGLANG_CMD=( ... --context-length "$MAX_MODEL_LEN" ... ) array literal. "$MAX_MODEL_LEN" under set -u → dsv4_fp4_mi355x_sglang.sh: line 149: MAX_MODEL_LEN: unbound variable. Script exits.
6. python3 -m sglang.launch_server is never invoked; wait_for_server_ready never runs; the sweep row logs a launcher failure with an empty server.log.

Blast radius

configs/amd-master.yaml dsv4-fp4-mi355x-sglang-agentic search-space has exactly four rows (all tp: 8, conc-list: [48]): TP8 baseline, TP8 + HiCache, TP8 + DPA, TP8 + DPA + HiCache. Every one of them routes through this launcher, so the entire recipe is 100% broken by this PR.

Fix

Drop the --context-length "$MAX_MODEL_LEN" line. This matches every other agentic launcher in the repo — none of them pass --context-length or --max-model-len, deferring to the model's native context (which is exactly the policy the unset in benchmark_lib.sh is enforcing). If a cap is genuinely needed later, use --context-length "${MAX_MODEL_LEN:-<agentic-default>}".

[claude]

🟡 The default arm of the KV_OFFLOAD_BACKEND case at line 85 prints (expected: native, mooncake, lmcache), but the case above only accepts hicache — the listed values look copy-pasted from the sibling vLLM launchers (which do accept native/mooncake/lmcache) and would themselves fall into the same default arm here. Sweep configs only use hicache so CI never trips this branch, but anyone debugging by hand will be misdirected. Fix: change the list to (expected: hicache).

Extended reasoning...

What the bug is

The case "${KV_OFFLOAD_BACKEND:-}" block at lines 56-88 of benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh has exactly one accepting arm: hicache). Anything else falls into the default *) arm at lines 84-87, which emits:

Error: unsupported KV_OFFLOAD_BACKEND '<value>' (expected: native, mooncake, lmcache)

None of native, mooncake, or lmcache are recognized by this case statement — they are all backends supported by other launchers in the repo (native for kimik2.5-fp4-mi355x-vllm-agentic, mooncake for minimaxm3-fp8-mi300x-vllm-agentic, lmcache for the sibling dsv4-fp4-mi355x-vllm-agentic). The message is a copy-paste from those vLLM-family launchers.

Why the message is self-contradictory

A user who sees unsupported KV_OFFLOAD_BACKEND 'foo' and reads the parenthetical hint will pick one of native, mooncake, or lmcache — and hit the exact same default arm and the exact same message. In the case where they picked native, the output literally reads unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache), which is self-contradictory.

Step-by-step proof

1. Operator invokes the launcher manually with KV_OFFLOADING=dram KV_OFFLOAD_BACKEND=native (a natural attempt after seeing another dsv4 launcher use lmcache).
2. agentic_kv_offload_enabled returns true because KV_OFFLOADING=dram.
3. Bash enters the case at line 56 and evaluates the arms in order. The only arm is hicache).
4. native does not match hicache), so control falls to *) at line 84.
5. Line 85 prints: Error: unsupported KV_OFFLOAD_BACKEND 'native' (expected: native, mooncake, lmcache).
6. Line 86 exits 1.
7. Operator, taking the message at face value, retries with mooncake or lmcache. Same path — steps 3-6 repeat.

Why existing code doesn't prevent it

The header comment at line 7 states "KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache", so the canonical documented value is hicache. The error text just wasn't updated to match when this launcher was adapted from the vLLM sibling.

Impact

Cosmetic misdirection in a diagnostic-only path. The four sweep entries in configs/amd-master.yaml for this recipe all pass kv-offload-backend: hicache, so CI never hits the default arm. The cost is a minute or two of wasted debug time for someone experimenting with a non-hicache backend by hand.

Fix

Change line 85's parenthetical to (expected: hicache):

echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: hicache)" >&2

On the refutations

Both refutations are duplicate-management: bug_002 and bug_004 target the same defect at the same lines with the same fix, and the refuters explicitly note the finding itself is real — they only refuted one of the two to avoid posting the same comment twice. The synthesis agent merged them into a single report, which resolves that concern.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29076727890
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29076727890

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29077198891
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29077198891

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081203249
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081203249

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081361033
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081361033