Add SageMaker hosting entrypoint to vLLM Neuron DLC 0.24.0.1.1.0

[malav-shastri]

Draft / for reference. Opening this to make the problem concrete and to offer one possible fix. Happy to close it if you'd rather implement this differently or as part of a separate SageMaker-flavored build — the useful part is the repro and the finding.

Problem

The published image can't be used as an Amazon SageMaker inference container. SageMaker launches one as docker run <image> serve, appending serve as an argument and overriding CMD. The current ENTRYPOINT (common/vllm_entrypoint.py) is a passthrough that execs its argv verbatim:

import subprocess, sys
subprocess.check_call(sys.argv[1:])

so serve is interpreted as a command to run, and there is no such executable in the image:

$ docker run --rm \
    public.ecr.aws/neuron/pytorch-inference-vllm-neuronx@sha256:be11c204f419a63e2487b2124005156dad091fb9edbfcadf42d81b745e284c12 \
    serve
FileNotFoundError: [Errno 2] No such file or directory: 'serve'

The container exits immediately, so CreateEndpoint fails at launch. (Digest above is the current 0.24.0.1.1.0-neuronx-py313-sdk2.32.0-ubuntu24.04 tag, built 2026-08-18.)

To be clear, this isn't a defect — the passthrough is the right design for EC2/Kubernetes, where the caller supplies the command. It's a mismatch with SageMaker's convention of passing the mode as argv[1].

Everything else SageMaker needs is already here

• Runs as root (Config.User empty)
• Port 8080 exposed, HEALTHCHECK already hits /ping
• /opt/ml/model created
• vLLM 0.24.0 registers /ping and /invocations natively via vllm/entrypoints/serve/sagemaker/api_router.py, called unconditionally from api_server.py
• model_hosting_container_standards 0.1.16 is installed, so standard-supervisor is on PATH at /opt/conda/bin/standard-supervisor

So only the process-launch step is missing.

Approach: add serve rather than change ENTRYPOINT

Because the entrypoint execs its argv, simply providing an executable named serve on PATH is enough. This means:

• docker run <image> serve → SageMaker works
• docker run <image> vllm serve ... → existing EC2/k8s usage unchanged

No ENTRYPOINT change, no new build stage, no behavior change for current users.

Changes

File
common/sagemaker_args.py	new — translates SM_VLLM_* env vars into vLLM CLI args
common/serve	new, 26 lines — builds args and execs the server under standard-supervisor
vllm/inference/0.24.0.1.1.0/Dockerfile.neuronx	copies both to /usr/local/bin (+6 lines)

Both files are adapted from aws/deep-learning-containers scripts/docker/vllm/ (Apache-2.0, same license as this repo), which is what the GPU SageMaker DLC uses. Two pieces of the GPU original are intentionally dropped:

• bash_telemetry.sh — not present in this image
• --middleware sagemaker_serve.SageMakerRouteMiddleware — unnecessary, since vLLM >= 0.24 registers the SageMaker routes natively

Verification

Built locally on top of the published image. Before:

FileNotFoundError: [Errno 2] No such file or directory: 'serve'

After, with SM_VLLM_MODEL=openai/gpt-oss-20b SM_VLLM_TENSOR_PARALLEL_SIZE=8 SM_VLLM_MAX_MODEL_LEN=8192:

INFO: vLLM server arguments: ['--port', '8080', '--max-model-len', '8192',
      '--model', 'openai/gpt-oss-20b', '--tensor-parallel-size', '8']
standard_supervisor: Starting: python3 -m vllm.entrypoints.openai.api_server ...
supervisord started with pid 10
INFO success: app entered RUNNING state
...
RuntimeError: Failed to infer device type

The env-to-CLI translation, supervisor launch, and server startup all succeed. The final error is the test host having no Trainium devices (vllm_neuron: No Neuron devices found. Skipping Neuron plugin registration), not a problem with the change. I have not yet run this on a trn2 instance — if someone on your side can, that would confirm the server comes up and serves /invocations end to end.

Two related notes, not in this diff

1. DLC labels. This image carries only dlc_major_version, maintainer, and org.opencontainers.image.version. The GPU SageMaker DLC carries the full com.amazonaws.ml.engines.sagemaker.dlc.* taxonomy (device.gpu.cu130, framework.vllm.0-29-0, python.py312, os.ubuntu24-04, arch.x86). Downstream tooling reads those to identify a DLC and its accelerator, and there's no Neuron device label today.
2. Regional publication. This image appears only on ECR Public. SageMaker hosting pulls from regional private ECR (or a VPC registry), and network-isolated endpoints can't reach a public registry at all, so a regionally-published image is required for any managed-catalog use. huggingface-vllm-inference-neuronx is already in 763104351884 across 11 regions, so the path exists — is putting the plugin line on it planned?

Context: evaluating this image for AWS Trainium support in Amazon SageMaker JumpStart, with openai/gpt-oss-20b as the pilot model.