fix: reclaim unused encrypted data disk blocks

CHANGELOG.md

@@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - http-client: a caller can bound the response body (`http_request_bounded`, `PrpcClient::with_max_response_bytes`). Nothing is bounded by default — `dstack vmm logs --lines 100000` is a legitimate multi-megabyte fetch — but every client that talks to a guest agent opts in, in the gateway and in the VMM, because a CVM is untrusted and one of them polls on a timer against the whole fleet
 
 ### Fixed
+- data disks: discard now propagates through ZFS or ext4, dm-crypt, virtio-blk, and QEMU so encrypted qcow2 images release deleted blocks instead of growing with lifetime writes. Discard defaults on and can be disabled with `storage_discard: false` when allocation-pattern leakage is unacceptable; upgrading an existing ZFS pool also starts a one-time trim for historical free space
 - certbot: a certificate covering both a name and its wildcard (`example.com` and `*.example.com`) could never be issued over dns-01. The two authorizations are answered under one `_acme-challenge.example.com`, each with its own TXT value, and the publish step cleared every TXT record at that name before writing its own -- so the second authorization deleted the record answering the first, and the order failed with `Correct value not found for DNS challenge`. Clearing leftovers from an aborted run is now done once per challenge name per issuance, and the records for one name accumulate instead of replacing each other; cleanup afterwards is unchanged, deleting each record this run created by id
 - certbot: editing `domains` in `certbot.toml` had no effect once a certificate existed. Issuance was skipped whenever `live/cert.pem` was present, whatever names it carried, and renewal read its name list back off that certificate rather than the configuration -- so an added or removed name never reached the CA, and the mismatch survived every renewal. The live certificate's DNS names are now compared against the configured list (as sets, case- and trailing-dot-insensitive) and a mismatch reissues, logging both lists. A reissue that fails does not take the renewal check down with it: a name the CA will not validate is reported on every cycle, while the certificate actually being served keeps renewing, and the failure is still what the run returns unless the renewal committed something of its own
 - certbot: `certbot cfg` attached every comment after `cf_api_url` to the wrong key, because an absent optional field shifts the generated document's keys out of step with the struct's. The template's documentation is now looked up by key name

docs/security/cvm-boundaries.md

@@ -43,6 +43,7 @@ This is the main configuration file for the application in JSON format:
 | pre_launch_script | 0.4.0 | string | Prelaunch bash script that runs before `docker compose up`. It runs *after* dockerd, so containers restored by a Docker restart policy can already be running when it executes. Do not build security gates on it — see [security-best-practices.md](./security-best-practices.md#security-semantics-must-not-depend-on-pre_launch_script-running-first). |
 | init_script | 0.5.5 (string), 0.6.0 (string[]) | string or string[] | Up to 5 Bash scripts executed in order prior to dockerd startup, so they always complete before any container starts, on every boot; a string is treated as a one-element array. Multiple scripts require string `manifest_version: "3"` so older guests fail closed. MrConfigV3 binds the hashes only for manifest v3. |
 | storage_fs | 0.5.5 | string | Filesystem type for the data disk of the CVM. Supported values: "zfs", "ext4". default to "zfs". **ZFS:** Checksums every block, so a modified block fails the read. **ext4:** Lower overhead and faster I/O for database workloads; checksums metadata but not file data, so a modified block is not detected. |
+| storage_discard | 0.6.0 | boolean | Return unused data-disk blocks to the host. Defaults to `true` so encrypted sparse images track live data instead of historical writes. Set to `false` if revealing filesystem allocation and deletion patterns to the host is unacceptable. |
 | swap_size | 0.5.5 | string/integer | The linux swap size. default to 0. Can be in byte or human-readable format (e.g., "1G", "256M"). |
 | key_provider | 0.5.6 | string | Key provider type. Supported values: "none", "kms", "local", "tpm". GCP vTPM and AWS EC2 NitroTPM are part of their platform trust models. The Dstack platform can use VMM-managed swtpm for seal/unseal and restart persistence, but it offers no protection against the host and is intentionally not accepted by remote verifiers. |
 

dstack/dstack-types/src/lib.rs

@@ -253,6 +253,10 @@ pub struct AppCompose {
     pub secure_time: bool,
     #[serde(default)]
     pub storage_fs: Option<String>,
+    /// Return unused data-disk blocks to the host. Disable this when leaking
+    /// filesystem allocation and deletion patterns is unacceptable.
+    #[serde(default = "default_true")]
+    pub storage_discard: bool,
     #[serde(default, with = "human_size")]
     pub swap_size: u64,
     #[serde(default, skip_serializing_if = "EventLogVersion::is_v1")]
@@ -825,6 +829,20 @@ mod app_compose_tests {
         }))
     }
 
+    #[test]
+    fn storage_discard_defaults_on_and_can_be_disabled() {
+        assert!(parse_compose(serde_json::json!(2)).unwrap().storage_discard);
+
+        let compose: AppCompose = serde_json::from_value(serde_json::json!({
+            "manifest_version": 2,
+            "name": "test",
+            "runner": "docker-compose",
+            "storage_discard": false
+        }))
+        .unwrap();
+        assert!(!compose.storage_discard);
+    }
+
     #[test]
     fn init_script_accepts_string_array_and_null() {
         assert!(parse_compose(serde_json::json!(2))
@@ -2637,6 +2655,7 @@ mod appcompose_sdk_parity {
             "requirements",
             "runner",
             "secure_time",
+            "storage_discard",
             "storage_fs",
             "swap_size",
             "verity_volumes",
@@ -2651,6 +2670,7 @@ mod appcompose_sdk_parity {
             "docker_compose_file": "services: {}\n",
             "init_script": ["a.sh"],
             "storage_fs": "ext4",
+            "storage_discard": false,
             "swap_size": "2G",
             "event_log_version": 2,
             "port_policy": {"ports": [{"port": 8080, "pp": true}], "restrict_mode": true},

dstack/dstack-util/src/system_setup.rs

@@ -160,6 +160,7 @@ impl FromStr for FsType {
 struct DstackOptions {
     storage_encrypted: bool,
     storage_fs: FsType,
+    storage_discard: bool,
 }
 
 fn parse_dstack_options(shared: &HostShared) -> Result<DstackOptions> {
@@ -168,6 +169,7 @@ fn parse_dstack_options(shared: &HostShared) -> Result<DstackOptions> {
     let mut options = DstackOptions {
         storage_encrypted: true, // Default to encryption enabled
         storage_fs: FsType::Zfs, // Default to ZFS
+        storage_discard: true,   // Reclaim unused blocks from sparse host images
     };
 
     for param in cmdline.split_whitespace() {
@@ -187,6 +189,7 @@ fn parse_dstack_options(shared: &HostShared) -> Result<DstackOptions> {
     if let Some(fs) = &shared.app_compose.storage_fs {
         options.storage_fs = fs.parse().context("Failed to parse storage_fs")?;
     }
+    options.storage_discard = shared.app_compose.storage_discard;
     Ok(options)
 }
 
@@ -2680,27 +2683,31 @@ impl<'a> Stage0<'a> {
 
             if opts.storage_encrypted {
                 info!("Setting up disk encryption");
-                self.luks_setup(disk_crypt_key, name)?;
+                self.luks_setup(disk_crypt_key, name, opts.storage_discard)?;
             } else {
                 info!("Skipping disk encryption as requested by kernel cmdline");
             }
 
             match opts.storage_fs {
                 FsType::Zfs => {
                     info!("Creating ZFS filesystem");
+                    let autotrim = if opts.storage_discard { "on" } else { "off" };
                     cmd! {
-                        zpool create -o autoexpand=on -m none dstack $fs_dev;
+                        zpool create -o autoexpand=on -o autotrim=$autotrim -m none dstack $fs_dev;
                         zfs create -o mountpoint=$mount_point -o atime=off -o checksum=blake3 dstack/data;
                     }
                     .context("Failed to create zpool")?;
                 }
                 FsType::Ext4 => {
                     info!("Creating ext4 filesystem");
-                    cmd! {
-                        mkfs.ext4 -F $fs_dev;
-                        mount $fs_dev $mount_point;
+                    cmd!(mkfs.ext4 -F $fs_dev).context("Failed to create ext4 filesystem")?;
+                    if opts.storage_discard {
+                        cmd!(mount -o discard $fs_dev $mount_point)
+                            .context("failed to mount ext4 filesystem with discard")?;
+                    } else {
+                        cmd!(mount $fs_dev $mount_point)
+                            .context("failed to mount ext4 filesystem")?;
                     }
-                    .context("Failed to create ext4 filesystem")?;
                 }
             }
         } else {
@@ -2710,7 +2717,7 @@ impl<'a> Stage0<'a> {
 
             if opts.storage_encrypted {
                 info!("Mounting encrypted data disk");
-                self.open_encrypted_volume(disk_crypt_key, name)?;
+                self.open_encrypted_volume(disk_crypt_key, name, opts.storage_discard)?;
             } else {
                 info!("Mounting unencrypted data disk");
             }
@@ -2719,24 +2726,44 @@ impl<'a> Stage0<'a> {
                 FsType::Zfs => {
                     cmd! {
                         zpool import dstack;
+                    }
+                    .context("Failed to import zpool")?;
+                    let previous_autotrim = cmd!(zpool get -H -o value autotrim dstack)
+                        .map(|value| value.trim().to_owned())
+                        .unwrap_or_default();
+                    let autotrim = if opts.storage_discard { "on" } else { "off" };
+                    cmd! {
+                        zpool set autotrim=$autotrim dstack;
                         zpool status dstack;
                         zpool online -e dstack $fs_dev; // triggers autoexpand
                     }
-                    .context("Failed to import zpool")?;
+                    .context("Failed to configure zpool")?;
+                    if opts.storage_discard && previous_autotrim == "off" {
+                        // autotrim only covers future frees. Start an asynchronous
+                        // trim on first upgrade so historical free space is returned
+                        // without delaying boot.
+                        if let Err(err) = cmd!(zpool trim dstack) {
+                            warn!("failed to start initial zpool trim: {err}");
+                        }
+                    }
                     if cmd!(mountpoint -q $mount_point).is_err() {
                         cmd!(zfs mount dstack/data).context("Failed to mount zpool")?;
                     }
                 }
                 FsType::Ext4 => {
-                    Self::mount_e2fs(&fs_dev, mount_point)
+                    Self::mount_e2fs(&fs_dev, mount_point, opts.storage_discard)
                         .context("Failed to mount ext4 filesystem")?;
                 }
             }
         }
         Ok(())
     }
 
-    fn mount_e2fs(dev: &impl AsRef<Path>, mount_point: &impl AsRef<Path>) -> Result<()> {
+    fn mount_e2fs(
+        dev: &impl AsRef<Path>,
+        mount_point: &impl AsRef<Path>,
+        discard: bool,
+    ) -> Result<()> {
         let dev = dev.as_ref();
         let mount_point = mount_point.as_ref();
         info!("Checking filesystem");
@@ -2764,17 +2791,23 @@ impl<'a> Stage0<'a> {
             }
         }
 
-        cmd! {
-            info "Trying to resize filesystem if needed";
-            resize2fs $dev;
-            info "Mounting filesystem";
-            mount $dev $mount_point;
+        if discard {
+            cmd! {
+                resize2fs $dev;
+                mount -o discard $dev $mount_point;
+            }
+            .context("failed to resize and mount ext4 filesystem with discard")?;
+        } else {
+            cmd! {
+                resize2fs $dev;
+                mount $dev $mount_point;
+            }
+            .context("failed to resize and mount ext4 filesystem")?;
         }
-        .context("Failed to prepare ext4 filesystem")?;
         Ok(())
     }
 
-    fn luks_setup(&self, disk_crypt_key: &str, name: &str) -> Result<()> {
+    fn luks_setup(&self, disk_crypt_key: &str, name: &str, discard: bool) -> Result<()> {
         let root_hd = &self.args.device;
         let sector_offset = PAYLOAD_OFFSET / 512;
         info!("Formatting encrypted disk");
@@ -2810,10 +2843,10 @@ impl<'a> Stage0<'a> {
         {
             bail!("Failed to setup luks volume");
         }
-        self.open_encrypted_volume(disk_crypt_key, name)
+        self.open_encrypted_volume(disk_crypt_key, name, discard)
     }
 
-    fn open_encrypted_volume(&self, disk_crypt_key: &str, name: &str) -> Result<()> {
+    fn open_encrypted_volume(&self, disk_crypt_key: &str, name: &str, discard: bool) -> Result<()> {
         let root_hd = &self.args.device;
         let disk_crypt_key = disk_crypt_key.trim();
         // Create a private tmpfs mount to ensure the header stays in-memory.
@@ -2841,8 +2874,13 @@ impl<'a> Stage0<'a> {
         validate_luks2_headers(hdr_file).context("Failed to validate LUKS2 header")?;
 
         info!("Opening the device");
-        let mut child = Command::new("cryptsetup")
-            .args(["luksOpen", "--type", "luks2", "--header"])
+        let mut command = Command::new("cryptsetup");
+        command.args(["luksOpen", "--type", "luks2"]);
+        if discard {
+            command.arg("--allow-discards");
+        }
+        let mut child = command
+            .arg("--header")
             .arg(&in_mem_hdr)
             .arg("-d-")
             .arg(root_hd)

dstack/guest-agent/src/rpc_service.rs

@@ -1151,6 +1151,7 @@ pub(crate) mod tests {
             no_instance_id: false,
             secure_time: false,
             storage_fs: None,
+            storage_discard: true,
             swap_size: 0,
             event_log_version: EventLogVersion::V1,
             port_policy: Default::default(),

dstack/vmm/src/app/qemu.rs

@@ -184,6 +184,7 @@ struct PreparedQemuLaunch {
     platform: CvmPlatform,
     networks: Vec<Networking>,
     volumes: Vec<PreparedVolume>,
+    storage_discard: bool,
     hugepage_numa_nodes: Option<HashMap<String, u32>>,
     gpu_numa_nodes: HashMap<String, String>,
     numa_cpus: Option<String>,
@@ -281,6 +282,7 @@ impl PreparedQemuLaunch {
             platform,
             networks,
             volumes,
+            storage_discard: app_compose.storage_discard,
             hugepage_numa_nodes,
             gpu_numa_nodes,
             numa_cpus,
@@ -558,8 +560,13 @@ impl QemuCommandBuilder<'_> {
         command
             .arg("-drive")
             .arg(format!(
-                "file={},if=none,id=hd1",
-                self.prepared.workdir.hda_path().display()
+                "file={},if=none,id=hd1,discard={}",
+                self.prepared.workdir.hda_path().display(),
+                if self.prepared.storage_discard {
+                    "unmap"
+                } else {
+                    "ignore"
+                }
             ))
             .arg("-device")
             .arg(virtio_pci_device(
@@ -1158,6 +1165,7 @@ mod tests {
             volumes: vec![PreparedVolume {
                 source: "/does-not-exist/volume.img".into(),
             }],
+            storage_discard: true,
             hugepage_numa_nodes: None,
             gpu_numa_nodes: HashMap::new(),
             numa_cpus: None,
@@ -1190,6 +1198,9 @@ mod tests {
             .args
             .windows(2)
             .any(|args| args == ["-append", "console=hvc0"]));
+        assert!(process.args.iter().any(|arg| {
+            arg == "file=/does-not-exist/vm-1/hda.img,if=none,id=hd1,discard=unmap"
+        }));
         assert!(process.args.windows(2).any(|args| {
             args == [
                 "-drive",
@@ -1231,6 +1242,19 @@ mod tests {
             .iter()
             .any(|arg| arg.contains("virtio-net-pci,netdev=net1")));
 
+        prepared.storage_discard = false;
+        let process = QemuCommandBuilder {
+            vm: &vm,
+            cfg: &config.cvm,
+            gpus: &GpuConfig::default(),
+            prepared: &prepared,
+        }
+        .build()
+        .unwrap();
+        assert!(process.args.iter().any(|arg| {
+            arg == "file=/does-not-exist/vm-1/hda.img,if=none,id=hd1,discard=ignore"
+        }));
+
         for network in &mut prepared.networks {
             network.mode = NetworkingMode::Bridge;
             network.bridge = "br0".into();

dstack/vmm/ui/src/components/CreateVmDialog.ts

@@ -90,6 +90,14 @@ const CreateVmDialogComponent = {
               </select>
             </div>
 
+            <div class="form-group checkbox-group">
+              <label>
+                <input v-model="form.storage_discard" type="checkbox">
+                Reclaim unused storage blocks
+                <span class="help-icon" title="Keeps sparse disk images small, but reveals allocation and deletion patterns to the host.">?</span>
+              </label>
+            </div>
+
             <div class="form-group full-width">
               <label for="appId">App ID (optional)</label>
               <input id="appId" v-model="form.app_id" type="text" placeholder="Leave empty for automatic generation">

dstack/vmm/ui/src/composables/useVmManager.ts

@@ -28,6 +28,7 @@ type AppCompose = {
   secure_time: boolean;
   requirements?: Requirements;
   storage_fs?: string;
+  storage_discard?: boolean;
   swap_size: number;
   launch_token_hash?: string;
   pre_launch_script?: string;
@@ -132,6 +133,7 @@ type VmFormState = {
   ports: PortFormEntry[];
   encryptedEnvs: EncryptedEnvEntry[];
   storage_fs: string;
+  storage_discard: boolean;
   app_id: string | null;
   key_provider?: KeyProviderKind;
   key_provider_id: string;
@@ -221,6 +223,7 @@ function createVmFormState(preLaunchScript: string): VmFormState {
     ports: [],
     encryptedEnvs: [],
     storage_fs: '',
+    storage_discard: true,
     app_id: null,
     key_provider: 'kms',
     key_provider_id: '',
@@ -876,6 +879,7 @@ type CreateVmPayloadSource = {
       allowed_envs: vmForm.value.encryptedEnvs.map((env) => env.key),
       no_instance_id: !vmForm.value.gateway_enabled,
       secure_time: false,
+      storage_discard: vmForm.value.storage_discard,
     };
 
     if (vmForm.value.key_provider !== undefined) {
@@ -1291,6 +1295,7 @@ type CreateVmPayloadSource = {
       encryptedEnvs: [], // Clear environment variables
       ports: [], // Clear port mappings
       storage_fs: theVm.appCompose?.storage_fs || 'zfs',
+      storage_discard: theVm.appCompose?.storage_discard ?? true,
       app_id: config.app_id || '',
       kms_urls: config.kms_urls || [],
       key_provider: getKeyProvider(theVm),

sdk/go/dstack/compose_hash.go

@@ -120,6 +120,7 @@ type AppCompose struct {
 	Requirements            *Requirements   `json:"requirements,omitempty"`
 	InitScript              []string        `json:"init_script,omitempty"`
 	StorageFs               string          `json:"storage_fs,omitempty"`
+	StorageDiscard          *bool           `json:"storage_discard,omitempty"`
 	// SwapSize is a human size string, e.g. "2G", matching what the guest reads.
 	SwapSize        string         `json:"swap_size,omitempty"`
 	EventLogVersion *uint32        `json:"event_log_version,omitempty"`

sdk/js/src/get-compose-hash.ts

@@ -116,6 +116,8 @@ export interface AppCompose extends SortableObject {
   requirements?: Requirements;
   init_script?: string[];
   storage_fs?: string;
+  /** Reclaim unused data-disk blocks; disable to hide allocation changes. */
+  storage_discard?: boolean;
   /** Human size string, e.g. "2G", matching what the guest reads. */
   swap_size?: string;
   event_log_version?: number;