feat(transaction): add unreferenced_files detection for expire-snapshots - #3038
Open
dhruvarya-db wants to merge 2 commits into
Open
feat(transaction): add unreferenced_files detection for expire-snapshots#3038dhruvarya-db wants to merge 2 commits into
dhruvarya-db wants to merge 2 commits into
Conversation
Add `iceberg::transaction::unreferenced_files`, which computes the files reachable only from a set of expiring snapshots — manifest lists, manifests, data/delete files, and statistics — so a caller can delete them after the expire-snapshots metadata commit. It mirrors Java `ReachableFileCleanup`: the result is the reference-count difference `files(expired) - files(retained)`, so a file still held by any retained snapshot is never returned. Data and delete files are only collected when `gc.enabled` is set, matching `drop_table_data`, since they can be shared across tables. Manifests are read through `ManifestFile::load_manifest`, so encrypted manifests are handled transparently. Any manifest-list or manifest read failure aborts the call, rather than risking deletion of a live file. This is metadata analysis only; nothing is deleted here. A follow-up will wire it into the commit path to perform the actual file deletion.
dhruvarya-db
marked this pull request as draft
August 20, 2026 21:14
Only live (added/existing) entries reference a file. A retained snapshot's manifest can carry a data file as a Deleted entry, and counting that as reachable made the anti-join treat the file as still live — so a file that was deleted by a retained snapshot but had been live in an expired one was never reported for deletion. Filter to `is_alive()` entries, matching Java's `liveEntries()` and the scan planner's handling.
dhruvarya-db
marked this pull request as ready for review
August 20, 2026 21:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #2145.
The metadata side of expire-snapshots is already in place from three merged PRs: #2591 added the
ExpireSnapshotsAction, #2664 made it honour thehistory.expire.*table properties by default, and #2667 removed statistics-file metadata for the expired snapshots. Those all editmetadata.json(dropping snapshots and their statistics) but none of them delete anything from storage, so the manifest lists, manifests, and data files that only the expired snapshots referenced are left behind.What changes are included in this PR?
This PR adds the detection step that finds those files, without deleting anything.
iceberg::transaction::unreferenced_files(table, expired_snapshot_ids)reads the manifest lists and manifests of the expiring snapshots and returns the files reachable only from them, grouped by kind (manifest lists, manifests, data files, delete files, and statistics). It mirrors Java'sReachableFileCleanup: the result is the reference-count differencefiles(expired) - files(retained), so anything a surviving snapshot still points at is never returned. Content files (data and deletes) are only collected whengc.enabledis set, matchingdrop_table_data, since they can be shared across tables; the table-private metadata is always collected. Manifests are read throughManifestReader, so encrypted manifests are handled transparently.Any read failure aborts the whole call: a manifest list or manifest that can't be read — for a retained or an expired snapshot alike — leaves the reachable set incomplete, and continuing could delete a file that is still live. This matches Java's
ReachableFileCleanup.This is metadata analysis only — nothing is deleted here. A follow-up PR will wire this into the commit path and perform the actual file deletion.
Are these changes tested?