Core: Use transaction metadata for post-commit file cleanup - #17708
Merged
findepi merged 1 commit intoAug 21, 2026
Conversation
uros-b
approved these changes
Aug 19, 2026
Member
|
+1, LGTM |
ebyhr
approved these changes
Aug 19, 2026
anoopj
reviewed
Aug 19, 2026
anoopj
approved these changes
Aug 19, 2026
findepi
approved these changes
Aug 20, 2026
BaseTransaction resolved its newly committed snapshots through ops.current(), a catalog re-read that can be stale when the catalog caches table pointers. A missing snapshot made committedFiles return null, skipping clean-up of all uncommitted files and leaking one stale manifest list per commit retry attempt. The transaction's own committed metadata always contains the new snapshots, so resolve them there.
raunaqmorarka
force-pushed
the
transaction-post-commit-cleanup
branch
from
August 20, 2026 12:29
27b96e1 to
a5fb265
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BaseTransaction resolves its newly committed snapshots through
ops.current(), a catalog re-read that can return stale metadata when the catalog caches table pointers. When any new snapshot is missing from the stale view,committedFilesreturns null and clean-up of all uncommitted files is skipped ("Failed to load metadata for a committed snapshot, skipping clean-up"). With concurrent writers on a Glue catalog this fired on every commit and leaked one stale manifest list per commit retry attempt.The transaction's own committed metadata always contains the new snapshots, so resolve them there.
The new test serves stale metadata from
current()after a successful commit with injected retries. Without the fix it fails because no uncommitted files are ever deleted.