[AMD][AgentX] DeepSeek V4 multi-node Agentic framework and recipe / DeepSeek V4 多节点 Agentic 框架与配方 by ichbinblau · Pull Request #2127 · SemiAnalysisAI/InferenceX

ichbinblau · 2026-07-09T06:58:28Z

Summary

Add end-to-end agentic benchmarking for DeepSeek-V4-Pro FP4 on MI355X using multi-node disaggregated SGLang and HiCache host-DRAM KV offloading.

This PR now focuses on the recipe and runtime implementation:

add the DSV4 MI355X disaggregated Agentic benchmark entrypoint;
add the AIPerf trace-replay path and Slurm/container wiring;
configure SGLang disaggregated serving and HiCache host-pool settings;
register the model, benchmark configuration, runner launch behavior, and changelog trigger;
expose SGLang host-cache hit rate in the existing structured Agentic result aggregation;
retain only local patch code that does not have a merged upstream equivalent.

Dependency

Depends on #2141, which separately enables kv-offloading and kv-offload-backend propagation for multi-node Agentic sweep matrices. The shared utils/matrix_logic changes are intentionally not included in this PR.

Until #2141 is merged, the configuration in this PR declares kv-offloading: dram and kv-offload-backend: hicache, but current main still forces multi-node Agentic matrix entries to kv-offloading: none.

Patch policy

Patch and overlay code corresponding to merged upstream SGLang changes has been removed. The remaining patch artifacts are limited to code without a merged upstream equivalent, including the disaggregated prefill bootstrap desynchronization fix and the reference-only decode TP queue agreement patch.

Validation

Shell and Slurm scripts pass bash -n.
configs/amd-master.yaml and perf-changelog.yaml parse successfully.
PYTHONPATH=. pytest -q utils/agentic/aggregation/test_process_agentic_result.py — 26 passed.
Changed Python aggregation files compile successfully.
git diff --check passes.

中文说明

本 PR 为 MI355X 上的 DeepSeek-V4-Pro FP4 增加端到端 Agentic 基准测试能力，采用多节点 SGLang 分离式推理，并通过 HiCache 使用主机 DRAM 进行 KV 缓存卸载。

本 PR 当前仅聚焦配方和运行时实现：

新增 DSV4 MI355X 分离式 Agentic 基准测试入口；
增加 AIPerf trace replay 路径以及 Slurm/容器运行链路；
配置 SGLang 分离式服务和 HiCache 主机缓存池参数；
注册模型、基准测试配置、runner 启动行为和 changelog 触发项；
在现有 Agentic 结构化结果聚合中暴露 SGLang 主机缓存命中率；
仅保留尚无已合并上游实现的本地补丁代码。

依赖关系

本 PR 依赖 #2141。#2141 单独为多节点 Agentic 扫描矩阵增加 kv-offloading 和 kv-offload-backend 的透传能力。本 PR 已移除所有共享的 utils/matrix_logic 改动。

在 #2141 合并之前，本 PR 的配置虽然声明了 kv-offloading: dram 和 kv-offload-backend: hicache，但当前 main 仍会把多节点 Agentic 矩阵项强制转换为 kv-offloading: none。

补丁策略

与已合并 SGLang 上游改动对应的补丁和 overlay 代码均已移除。当前仅保留尚无已合并上游等价实现的本地补丁，包括分离式 prefill bootstrap 不同步修复，以及仅作参考、默认不启用的 decode TP queue agreement 补丁。

验证

Shell 和 Slurm 脚本均通过 bash -n；configs/amd-master.yaml 与 perf-changelog.yaml 可正常解析；Agentic 聚合测试 26 项全部通过；相关 Python 文件编译检查和 git diff --check 均通过。

github-actions · 2026-07-09T06:58:37Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

functionstackx · 2026-07-09T07:38:35Z

      - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: native, conc-list: [24, 32, 36, 40, 44, 48, 52, 56, 60, 64, 72, 80, 96] }
+
+dsv4-fp4-mi355x-sglang-disagg-agentic-hicache:
+  image: rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260706


do u have upstream image @ichbinblau

We have bumped the image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260706

Yes we will switch to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 where included all 4 patches

Duyi-Wang · 2026-07-09T10:51:50Z

The patches in this PR are derived from the following upstream SGLang PRs.
sgl-project/sglang#30374
sgl-project/sglang#30333
sgl-project/sglang#30339
sgl-project/sglang#29417

github-actions · 2026-07-09T21:50:37Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29052264020
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29052264020

github-actions · 2026-07-09T21:52:44Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29052670710
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29052670710

github-actions · 2026-07-09T22:19:12Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29054064553
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29054064553

github-actions · 2026-07-09T23:12:24Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29056644716
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29056644716

github-actions · 2026-07-09T23:21:56Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29057082376
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29057082376

github-actions · 2026-07-09T23:26:29Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29057275526
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29057275526

cquil11 · 2026-07-10T00:24:34Z

+         )
+         kv_manager._staging_handler = self.staging_handler
+
+    def _agree_and_order_queue(self) -> Tuple[List["DecodeRequest"], List["DecodeRequest"]]:


We will not accept any patches. I have removed all patches that have since been merged into upstream sglang main. Please update nightly image tomorrow.

However, this patch remains. Please remove it / open upstream pr.

We will not accept any patches. I have removed all patches that have since been merged into upstream sglang main. Please update nightly image tomorrow.

However, this patch remains. Please remove it / open upstream pr.

This patch file has not been used as its caller func has been commented out. I have remove it.

cquil11 · 2026-07-10T00:25:33Z

@ichbinblau Thank you for this awesome work. Please see the changes I requested. Also, please re run full-sweep-enabled after making these changes.

+viz @chunfangamd

matrix.config.conc for multi-node agentic is already a JSON array (e.g. [16,32,64] from generate_sweep_configs.py). Wrapping it in a string literal as '[${{ matrix.config.conc }}]' can't correctly interpolate an array into a string, producing a malformed conc-list so fromJson(inputs.conc-list) in the reusable template ended up with an empty/wrong CONC_LIST. Use toJson(matrix.config.conc) instead, matching the pattern already used for the other two conc-list inputs in this file. Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Theresa Shan <theresa.shan@amd.com>

github-actions · 2026-07-10T03:05:11Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29065983132
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29065983132

matrix.config.conc for multi-node agentic entries is a JSON array (chunked concurrencies per allocation), but sweep-multi-node-agentic passed it directly into benchmark-multinode-tmpl.yml's `conc` input, which is declared `type: string` ("First concurrency for agentic-coding scenarios; CONC_LIST carries the full batch"). GitHub Actions' reusable-workflow input validator rejects a sequence value for a string-typed input at evaluation time, so the whole job failed to even load: evaluate reusable workflow inputs: .github/workflows/run-sweep.yml (Line: 554, Col: 19): A sequence was not expected Since the job never materializes when this happens, sweep-multi-node- agentic silently disappeared from run summaries entirely instead of showing as failed. Slice to the first element (matching the intended "first concurrency" semantics and the same fix already applied elsewhere, e.g. PR #2122) to restore a scalar value. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-10T03:18:24Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066428461
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29066428461

github-actions · 2026-07-10T03:22:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066543853
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29066543853

Multi-node agentic sweeps batched up to 4 concurrencies per SLURM allocation, running them sequentially against one shared server session. A slow/hung conc could block the rest of the batch from ever producing results, which is why run #6719 only reported c16 despite a 16/32/48/64 conc-list. Drop the batch size to 1 so each concurrency gets its own task/allocation, matching the granularity already used for single-node agentic sweeps. Cherry-picked from backup/agentx-v1.0-rebase-pre-upstream-rewrite-20260710 (082a59d), adapted for the current test suite. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-10T03:38:53Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066685174
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29066685174

github-actions · 2026-07-10T07:40:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29067272731
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29067272731

github-actions · 2026-07-10T10:53:42Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29078166951
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29078166951

Signed-off-by: Theresa Shan <theresa.shan@amd.com>

patch_decode_tp_queue_agree()'s invocation was already commented out (dead code with no runtime effect), and the reference-only patches/decode_tp_queue_agree.patch it mirrored had no other callers. Drop both plus the README bullet pointing at it. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-07-10T11:24:55Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29078166951
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29078166951

ichbinblau requested a review from a team July 9, 2026 06:58

ichbinblau requested review from billishyahao, chunfangamd, seungrokj and yctseng0211 as code owners July 9, 2026 06:58

github-project-automation Bot added this to InferenceMAX Board Jul 9, 2026

ichbinblau requested a review from 1am9trash as a code owner July 9, 2026 06:58

billishyahao added AMD full-sweep-fail-fast labels Jul 9, 2026

claude Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread utils/matrix_logic/validation.py Outdated

Comment thread .github/workflows/run-sweep.yml Outdated

seungrokj changed the title ~~Deepseek V4 Multi-node Agentic Framework and Recipe~~ [AMD][AgentX] Deepseek V4 Multi-node Agentic Framework and Recipe Jul 9, 2026

functionstackx reviewed Jul 9, 2026

View reviewed changes

ichbinblau added full-sweep-enabled and removed full-sweep-fail-fast labels Jul 9, 2026

ichbinblau force-pushed the amd/agentx-v1.0-rebase branch from 3f0a1f9 to da1084d Compare July 9, 2026 14:11

cquil11 added the agentx AgentX benchmarks, recipes, and infrastructure label Jul 9, 2026 — with ChatGPT Codex Connector

cquil11 force-pushed the amd/agentx-v1.0-rebase branch 2 times, most recently from 86da4cc to a34c27e Compare July 9, 2026 21:50

cquil11 force-pushed the amd/agentx-v1.0-rebase branch from a34c27e to b0088c8 Compare July 9, 2026 22:16

cquil11 force-pushed the amd/agentx-v1.0-rebase branch from b0088c8 to 4e557d0 Compare July 9, 2026 23:10

cquil11 changed the title ~~[AMD][AgentX] Deepseek V4 Multi-node Agentic Framework and Recipe~~ [AMD][AgentX] DeepSeek V4 multi-node Agentic framework and recipe / DeepSeek V4 多节点 Agentic 框架与配方 Jul 9, 2026

cquil11 force-pushed the amd/agentx-v1.0-rebase branch from 4e557d0 to fce18af Compare July 9, 2026 23:20

[AMD][AgentX] Add DeepSeek V4 MI355X agentic disagg

b6a9877

cquil11 force-pushed the amd/agentx-v1.0-rebase branch from fce18af to b6a9877 Compare July 9, 2026 23:24

cquil11 requested changes Jul 10, 2026

View reviewed changes

ichbinblau and others added 2 commits July 10, 2026 02:31

bump image to rocm/sgl-dev:sglang-0.5.14-rocm720-mi35x-mori-0706

82b4eff

Signed-off-by: Theresa Shan <theresa.shan@amd.com>

ichbinblau force-pushed the amd/agentx-v1.0-rebase branch from e9aa46a to 55a7983 Compare July 10, 2026 03:21

Bump image to latest upstream image with 4 PR fixes included.

c96cf81

ichbinblau and others added 2 commits July 10, 2026 11:23

update con-list

8040648

Signed-off-by: Theresa Shan <theresa.shan@amd.com>

Uh oh!

Conversation

ichbinblau commented Jul 9, 2026 • edited by cquil11 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependency

Patch policy

Validation

中文说明

依赖关系

补丁策略

验证

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

functionstackx Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

ichbinblau Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

billishyahao Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Duyi-Wang commented Jul 9, 2026 • edited by ichbinblau Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 9, 2026

Uh oh!

cquil11 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

ichbinblau Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cquil11 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ichbinblau commented Jul 9, 2026 •

edited by cquil11

Loading

ichbinblau Jul 9, 2026 •

edited

Loading

Duyi-Wang commented Jul 9, 2026 •

edited by ichbinblau

Loading

ichbinblau Jul 10, 2026 •

edited

Loading

cquil11 commented Jul 10, 2026 •

edited

Loading