feat(slurm): harden distributed allocation runtime

[nabinchha]

📋 Summary

This completes the distributed-topology and failure-hardening slice of the Slurm allocation runtime on top of the merged reconciliation/retry foundation.

The branch has been restacked directly on feat/slurm-execution at 36864772, which includes merged PRs #913, #915, #916, #929, and #930. The four PR-owned commits layer only host placement, multi-node launch, remote readiness, follower-failure behavior, model-path mapping and preflight, and a base-integration test adaptation onto that composition.

🔗 Related Issue

• Part of Implement Slurm submission and allocation runtime #868 (868#3: distributed execution and failure hardening)
• Depends on the merged shared Slurm base through fix: harden Slurm release artifacts #916

🔄 Changes

• resolve scheduler allocation hosts in planner order and pin client, endpoint, server, and control steps to their assigned hosts
• preserve the one-node per-process launch path while adding coordinated one-task-per-node srun steps for multi-node deployments
• add bounded node-worker specifications with deterministic GPU, rank, rendezvous, stagger, port ownership, and an optional validated model path
• preflight ports and mapped absolute model-path accessibility on every assigned deployment node before launch, and route readiness checks and logical endpoints to remote backend hosts
• propagate follower or lane failure across a deployment with sibling cleanup and --kill-on-bad-exit=1
• separate validated manifest records, server composition, distributed command construction, and node supervision into focused public modules
• map absolute model paths through the resolved container mount for coordinated and remote-node commands
• select vLLM's single-process executor only for TP×PP=1; use its multi-process executor whenever either parallel dimension spans workers
• preserve the merged retry-plan binding and effective-resume arguments while composing multi-node runtime setup

🧪 Testing

• 25 passed — focused bootstrap, entrypoint, node-worker, and shell-runtime tests after restacking
• 1,548 passed — full packages/data-designer-slurm/tests suite after restacking
• make check-slurm
• Slurm public-artifact audit passed for all 9 targets
• git diff --check
• Earlier branch validation built the data-designer-slurm wheel, verified the distributed runtime modules and shell assets, installed it in isolation, and imported the public runtime types
• real-cluster multi-node E2E proof remains in the joint Slurm acceptance lane

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• New behavior has focused regression coverage
• Architecture docs updated (not applicable; this implementation follows the reviewed Implement Slurm submission and allocation runtime #868 ownership boundary)

---

Description updated with AI

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issue or outstanding blocking finding remains.

Findings

1.  Sampler failure disables backpressure ▶

Fix with agent prompt

### Issue 1
packages/data-designer-slurm/src/data_designer/slurm/runtime/backpressure.py:123-126
If a vLLM or Prometheus metrics collector raises during sampling, the exception terminates this daemon thread while `_thread` remains non-`None`, so later requests cannot restart it. Once the cached snapshot becomes stale, admission permanently fails open and the configured queue limit stops producing 429 responses.

```suggestion
    def _sample_forever(self) -> None:
        while True:
            try:
                self.sample_once()
            except Exception:
                pass
            time.sleep(self.settings.poll_interval_seconds)
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

• Resolves scheduler hosts into planner order and pins runtime steps to their assigned nodes.
• Adds coordinated multi-node vLLM launch, node-local preflight, remote readiness, and sibling-failure cleanup.
• Maps absolute model paths through container mounts and selects vLLM executors according to the resolved parallel topology.
• Runs the plugin-aware client worker in a fresh interpreter so verified dependency overlays activate before plugin imports.
• Adds public-artifact auditing, package licensing metadata, and extensive focused regression coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Slurm allocation entrypoint] --> B[Resolve ordered allocation hosts]
    B --> C[Build validated runtime manifest]
    C --> D[Client preflight on client host]
    C --> E[Server preflight on deployment nodes]
    E --> F[srun one task per assigned node]
    F --> G[Node workers launch vLLM lanes]
    G --> H[Remote backend readiness probes]
    H --> I[Endpoint proxy on client host]
    I --> J[Fresh client-worker process]
    J --> K[Generate and publish candidate output]
    G -. lane or follower failure .-> L[Terminate sibling processes]

Loading

_{Reviews (13) · Last reviewed commit: "fix distributed launch preflight"}

[greptile-apps]

Sampler failure disables backpressure

If a vLLM or Prometheus metrics collector raises during sampling, the exception terminates this daemon thread while _thread remains non-None, so later requests cannot restart it. Once the cached snapshot becomes stale, admission permanently fails open and the configured queue limit stops producing 429 responses.

Suggested change

	def _sample_forever(self) -> None:
	while True:
	self.sample_once()
	time.sleep(self.settings.poll_interval_seconds)
	def _sample_forever(self) -> None:
	while True:
	try:
	self.sample_once()
	except Exception:
	pass
	time.sleep(self.settings.poll_interval_seconds)

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/data-designer-slurm/src/data_designer/slurm/runtime/backpressure.py
Line: 120-123

Comment:
**Sampler failure disables backpressure**

If a vLLM or Prometheus metrics collector raises during sampling, the exception terminates this daemon thread while `_thread` remains non-`None`, so later requests cannot restart it. Once the cached snapshot becomes stale, admission permanently fails open and the configured queue limit stops producing 429 responses.

```suggestion
    def _sample_forever(self) -> None:
        while True:
            try:
                self.sample_once()
            except Exception:
                pass
            time.sleep(self.settings.poll_interval_seconds)
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nabinchha]

Fixed in 77933cc. QueueBackpressureController.sample_once now converts ordinary reader failures into a fresh unavailable snapshot, so admission fails open for that sample and the daemon continues polling; the next successful sample restores queue-limit rejection. BaseException is intentionally not caught, preserving process-control and shutdown signals. Added regression coverage for failure, fail-open behavior, recovery, and KeyboardInterrupt propagation. Validation: 5 focused tests and 1,235 full Slurm tests passed; check-slurm and focused strict complexity checks pass.

[andreatnvidia]

Could we handle PYTHONPATH explicitly here? It is valid in server.environment, but this assignment silently replaces it. A deployment using an approved mount for a custom parser or plugin will validate and then fail at startup because its module path disappears. Either prepend the runtime bundle path or reject PYTHONPATH during config validation, with a regression test for the collision.

[nabinchha]

Addressed in b1de388. The staged runtime bundle path is now prepended while the deployment-configured PYTHONPATH value is preserved unchanged behind it. Added a regression covering multiple configured plugin/parser entries. Validation: 19 focused runtime tests and all 1,273 Slurm tests pass; make check-slurm and git diff --check also pass.

[andreatnvidia]

urlsplit(...).hostname lowercases hostnames, while allowed_hosts keeps the spelling returned by scontrol. That makes an accepted host such as Compute-001 fail this check and prevents the proxy from starting. Normalizing both sides before comparison, plus a mixed-case test, should cover it.

[nabinchha]

Addressed in b1de388. Backend parsing now compares the URL hostname and scheduler allow-list using casefolded forms, while retaining the parsed canonical hostname for the connection. Added a mixed-case scheduler-host regression. Validation: 19 focused runtime tests and all 1,273 Slurm tests pass; make check-slurm and git diff --check also pass.

[andreatnvidia]

Just a couple more things I found in a final review, both around distributed startup behavior. Once those are addressed, I think this is good to go.

[andreatnvidia]

Looks good now. The final couple of issues are addressed, and the latest changes look solid. Approving!