[WIP] Add GB300 AgentX Qwen3.5 recipes

[csahithi]

• 9 Qwen3.5-397B-A17B-NVFP4 GB300 sglang AgentX recipes (agg + disagg pareto)
• configs/nvidia-master.yaml: qwen3.5-fp4-gb300-dynamo-sglang{,-agentic-agg,-agentic-disagg}
• perf-changelog.yaml: agentic pareto entry
• runners/launch_gb300-nv.sh: qwen3.5 launcher branch + dynamo-wheels cache

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28975134400
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28975134400

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🔴 configs/nvidia-master.yaml:10940-10969 — 9 new sweep entries in configs/nvidia-master.yaml (agentic-agg + agentic-disagg, lines 10940–11085) declare spec-decoding: none, but every one of the 9 corresponding recipe files enables NEXTN MTP (speculative-algorithm: NEXTN + speculative-num-steps: 3 + eagle-topk/num-draft-tokens; the recipe filenames also encode -mtp-). The engine runs MTP correctly because srtctl applies the recipe directly, but SPEC_DECODING propagates verbatim into RESULT_FILENAME (_spec-none_… in artifact names, benchmark-multinode-tmpl.yml:225) and into the aggregated result JSON's spec_decoding field (utils/agentic/aggregation/process_agentic_result.py:139), so downstream dashboards / Pareto comparisons will silently misclassify these MTP runs as non-MTP. Fix: change spec-decoding: none → spec-decoding: "mtp" on all 9 entries in this block (matches the file's existing MTP convention — every other qwen3.5/dsr1/dsv4 MTP entry uses mtp; the parallel dsv4 agentic entry using none is self-consistent because its vLLM recipes have no speculative-* fields).

  Extended reasoning...

  What the bug is

  All 9 newly added sweep entries under qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg and qwen3.5-fp4-gb300-dynamo-sglang-agentic-disagg (configs/nvidia-master.yaml lines 10943, 10957, 10986, 11000, 11014, 11029, 11044, 11059, 11074) set:

  spec-decoding: none

  But every one of the 9 corresponding recipe files under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/agentic/*.yaml enables NEXTN MTP:

  speculative-algorithm: NEXTN
  speculative-num-steps: 3
  speculative-eagle-topk: 1
  speculative-num-draft-tokens: 4

  The recipe filenames themselves also encode -mtp- (e.g. agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml, disagg-gb300-1p1d-tp4-tp4-c32-mtp-hicache-convaware-jid2212783.yaml).

  Why the engine still runs MTP but the metadata is wrong

  srtctl apply reads the recipe file directly, so the engine gets --speculative-algorithm NEXTN … on its command line and NEXTN MTP runs as intended. The spec-decoding field in nvidia-master.yaml is not a live engine setting — it feeds SPEC_DECODING into the workflow env, which has two downstream consumers, and both are silently corrupted by this mislabel:

  1. Artifact filenames — .github/workflows/benchmark-multinode-tmpl.yml:225 bakes _spec-${{ env.SPEC_DECODING }}_ directly into RESULT_FILENAME. So uploaded result JSONs land as …_spec-none_conc*.json despite the run actually using MTP.

  2. Aggregated result JSON body — utils/agentic/aggregation/process_agentic_result.py:139 writes "spec_decoding": os.environ.get("SPEC_DECODING", "none") verbatim into the aggregated result JSON. Downstream dashboards / Pareto plots that filter or facet on spec_decoding will silently misclassify these MTP runs as non-MTP.

  (Some verifiers also flagged utils/compare_results.py:51 reading result['spec_decoding'] for baseline lookup — worth double-checking, but the two consumers above are sufficient on their own.)

  Convention check

  Every other MTP recipe in nvidia-master.yaml (dsr1-mtp, dsv4-b200-vllm-mtp, qwen3.5 fixed-seq-len MTP tiers, etc.) uses spec-decoding: "mtp". The one other agentic entry that legitimately uses spec-decoding: none — dsv4-fp4-gb300-dynamo-vllm-agentic — is self-consistent because its vLLM recipes at benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/*.yaml contain no speculative-* fields. So this is a real inconsistency introduced by this PR, not the file's existing style.

  Step-by-step proof (agentic-agg conc=1)

  1. Sweep matrix expands the spec-decoding: none / conc-list: [1] entry into a GHA job with SPEC_DECODING=none, CONFIG_FILE=recipes/sglang/qwen3.5/gb300-fp4/agentic/agg-gb300-tp4-c1-mtp-hicache-jid2191933.yaml.
  2. srtctl apply -f $CONFIG_FILE reads the recipe, which contains speculative-algorithm: NEXTN + speculative-num-steps: 3 — the sglang server starts with NEXTN MTP enabled. ✅ Engine correct.
  3. benchmark-multinode-tmpl.yml:225 computes RESULT_FILENAME=…_spec-none_conc1_… from env.SPEC_DECODING=none. ❌ Artifact filename claims non-MTP.
  4. process_agentic_result.py:139 reads os.environ.get("SPEC_DECODING", "none") → "none" and writes "spec_decoding": "none" into the aggregated result JSON. ❌ Result body claims non-MTP.
  5. A downstream MTP-vs-non-MTP Pareto plot that groups by spec_decoding puts this run in the non-MTP bucket, distorting the frontier.

  Fix

  Mechanical 9-line change — replace spec-decoding: none with spec-decoding: "mtp" on all 9 new entries in this block (lines 10943, 10957, 10986, 11000, 11014, 11029, 11044, 11059, 11074). Nothing else in the recipe files needs to change; the change is purely to keep the master-config label truthful to what the recipe actually runs.

[claude]

🔴 The new perf-changelog entry for qwen3.5-fp4-gb300-dynamo-sglang-agentic-{agg,disagg} (perf-changelog.yaml:4366-4373) will block the changelog gate on two independent grounds: (1) it's missing the required pr-link field, and (2) it's inserted mid-file rather than appended to the end. Fix: move the 8-line block to the end of the file and add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121 (or the XXX placeholder while WIP).

Extended reasoning...

What breaks. utils/validate_perf_changelog.py (invoked from .github/workflows/run-sweep.yml on every PR push) enforces three independent checks that this new entry violates. Any one of them is enough to fail the changelog gate; the entry fails all three.

1. Missing pr-link — schema-level rejection. In utils/matrix_logic/validation.py:683, ChangelogEntry declares pr_link: str = Field(alias="pr-link") with no default and extra="forbid". parse_changelog (validate_perf_changelog.py:128-133) calls ChangelogEntry.model_validate(entry) on every entry; Pydantic raises ValidationError on the new entry, which is re-raised as ChangelogValidationError('… fails ChangelogEntry validation'). Even if that were bypassed, validate_added_pr_link (lines 144-160) rejects it a second time: str(entry.get('pr-link') or '') == '', which is neither in PR_LINK_PLACEHOLDERS = {'XXX', 'https://github.com/SemiAnalysisAI/InferenceX/pull/XXX'} nor equal to the canonical PR URL, so it raises ChangelogValidationError('new PR entry must use …').

2. Mid-file insertion — positional rejection. The new 8-line block sits at lines 4366-4373, between an existing pull/1931 entry and the pre-existing minimaxm3-fp4-mi355x-vllm-mtp entry — with ~30 more historical entries after it. compare_entries (lines 163-208) walks base_entries and head_entries in lockstep by index; at the insertion index, head_entries[i] is the new qwen3.5 entry but base_entries[i] is the pre-existing minimaxm3 entry, and without_pr_link differs, so it raises ChangelogValidationError('entry N changed; existing entries are immutable except for pr-link-only corrections').

3. Byte-level prefix check. validate_raw_change (lines 211-235) additionally requires head_raw.startswith(base_raw) when there are additions. A mid-file insertion breaks that even before compare_entries runs, raising 'appended entries changed historical perf-changelog.yaml bytes; restore the base file byte-for-byte and append at the end'. AGENTS.md line 78 codifies the same rule: 'new entries MUST be appended to the END, never inserted in the middle or prepended.'

Step-by-step proof.

1. CI runs python3 utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD.
2. parse_changelog reads perf-changelog.yaml and iterates entries. When it reaches the qwen3.5 entry, ChangelogEntry.model_validate({config-keys: [...], description: [...]}) fails because pr-link is required and absent → ChangelogValidationError → exit non-zero → the Validate perf-changelog check turns red → PR cannot merge. If pr-link were the only issue, adding pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121 at the end of the block would clear check 1.
3. Even then, validate_raw_change computes base_raw = git show origin/main:perf-changelog.yaml and head_raw = the current file; because ~30 entries follow the insertion point in head_raw but not at the same byte offset in base_raw, head_raw.startswith(base_raw) is false → raises. Only after moving the 8-line block to the file end (after the current last entry) does this pass.
4. compare_entries then compares base and head entries index-by-index; index-of-first-difference is now len(base_entries), the appended entry is treated as an addition, validate_added_pr_link re-validates the pr-link, and the gate passes.

Fix. Two edits:

• Move the qwen3.5-fp4-gb300-dynamo-sglang-agentic-{agg,disagg} entry (perf-changelog.yaml:4366-4373) to the end of perf-changelog.yaml, after the current last entry.
• Add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2121 to the moved block (or pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX for early WIP).

[csahithi]

Resolved both the above comments

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28977280171
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28977280171

[functionstackx]

started about discussion about AgentX MTP +viz @kedarpotdar-nv

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28977625116
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=28977625116

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055853595
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29055853595

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29056354557
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29056354557