Skip to content

fix(guest): measure gpu-attestation after TPM key provisioning - #1410

Merged
kvinwang merged 1 commit into
nextfrom
fix/gpu-tpm-unseal-pcr14
Sep 26, 2026
Merged

kvinwang merged 1 commit into
nextfrom
fix/gpu-tpm-unseal-pcr14

Conversation

@kvinwang

@kvinwang kvinwang commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Root cause

A GPU CVM using the TPM key provider cannot reboot. Every boot after the first fails with failed to unseal from TPM … TPM error: 0x0000099d (policy failure).

  • The TPM key provider seals its seed to SHA-256 PCRs 0, 2 and 14 on GCP, and to SHA-384 PCRs 4, 7, 8, 12 and 14 on AWS.
  • Every runtime event is extended into PCR14.
  • Since 1fbae7b (fix(guest): bind GPU attestation to measured app policy #789), the GPU gate extends gpu-attestation before the keys are requested. Its evidence_sha256 hashes nvattest output made with a fresh random nonce, so PCR14 at unseal can never equal PCR14 at seal.

Fix

The GPU gate (nvattest, policy, NVML checks, ready state) still runs and fails closed before key provisioning. Only the gpu-attestation event is now extended after the keys are provisioned, between boot-mr-done and key-provider. PCR14 at the seal/unseal point then contains only deterministic events.

Tradeoff:

  • The event still comes before system-ready, so a verifier that follows the documented rule ("exactly one pre-system-ready gpu-attestation, evidence_sha256 matches the boot-time bundle") keeps working. The payload schema does not change.
  • KMS no longer sees the event in the quote at key-request time. KMS never inspected it, and the gate itself runs in measured guest code.
  • The boot-time mrAggregated (the boot-mr-done cutoff) no longer includes per-boot GPU evidence, so for GPU CVMs it is stable across boots.
  • The docs listing the event order are updated.

Rejected alternative: unsealing before the gate. On AWS, PCR8 is only measured after boot-mr-done, and that option would also split key provisioning by provider.

Verification

GCP us-central1-a, a3-highgpu-1g (1x H100, CC on), TDX, SPOT, key provider tpm. Dev mkosi image built rootless from this branch plus #1409 (needed for rootless builds) and #1411 (masks systemd's SRK units; it touches neither PCR 0/2/14 nor dstack's TPM handles).

  • Boot 1: GPU attestation passed, no sealed seed found, generating new seed, and the LUKS data disk was formatted and mounted. A marker file was written to /dstack/persistent.
  • gcloud compute instances reset.
  • Boot 2: GPU attestation passed, then unsealed root key seed from TPM (PCR policy: sha256:0,2,14). disk_crypt_key, k256_key and env_crypt_key are identical to boot 1. The same data disk opened and the boot-1 marker is present. app_id, instance_id and compose_hash are unchanged.
  • On both boots, PCR14 replayed up to boot-mr-done is the same value (d04276ff…). The full event log replays to Info RTMR3, to RTMR3 in a fresh /v1/Attest quote and to TPM PCR14. There is exactly one gpu-attestation event, after boot-mr-done. SHA-256 of the /v1/Attest boot-time GPU evidence bundle equals its evidence_sha256.
  • Before this change, the same setup reboot-looped with 0x99d after reset.
  • Local checks: cargo clippy -p dstack-util -- -D warnings and cargo test -p dstack-util (114 passed).

The gpu-attestation event commits to nvattest output made with a fresh
random nonce, so its digest changes on every boot. It was extended into
the runtime register before the app keys were requested, and the TPM key
provider seals its seed to that register (SHA-256 PCR14 on GCP, SHA-384
PCR14 on AWS). The PCR14 value at unseal could therefore never match the
value at seal, and a GPU CVM with the TPM key provider failed every boot
after the first with a TPM policy error (0x99d).

Keep the GPU gate (nvattest, policy, NVML checks, ready state) before key
provisioning, so an unattested GPU still stops the boot before any key is
released, and extend only the gpu-attestation event after the keys are
provisioned, between boot-mr-done and key-provider. It still precedes
system-ready, so a verifier replaying the event log sees the same payload
bound to the quote.

Regression from 1fbae7b (#789).

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang merged commit 5e950d9 into next Sep 26, 2026
11 checks passed
@kvinwang
kvinwang deleted the fix/gpu-tpm-unseal-pcr14 branch September 26, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant