[lake/lance] Harden LanceLakeCommitter close() and commit() error paths#3571
Open
XuQianJin-Stars wants to merge 1 commit into
Open
[lake/lance] Harden LanceLakeCommitter close() and commit() error paths#3571XuQianJin-Stars wants to merge 1 commit into
XuQianJin-Stars wants to merge 1 commit into
Conversation
Two failure-side improvements to LanceLakeCommitter: 1. commit() now catches RuntimeException from LanceDatasetAdapter and rethrows it as an IOException with the original cause preserved and the dataset URI in the message. Previously, Lance JNI/runtime exceptions escaped raw, so the tiering orchestrator could not treat them uniformly with other IO failures and operators had no context about which dataset failed. 2. close() now catches IllegalStateException that Arrow's RootAllocator raises when child buffers were leaked, and rethrows it as an IOException with an actionable message identifying the source. The allocator is still closed on happy paths and its close() call is no longer skipped by upstream failures - the exact leak text is kept as the cause so that debugging remains straightforward. Add LanceLakeCommitterTest covering: - close() after construction with no work performed does not leak. - commit() against a non-existent dataset raises IOException with a RuntimeException cause. - getMissingLakeSnapshot() against a missing dataset can be called and the committer can still be closed cleanly afterwards.
9ad17c1 to
3147c2d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two failure-side improvements to
LanceLakeCommitter:commit()now catchesRuntimeExceptionfromLanceDatasetAdapterand rethrows it as anIOExceptionwith the original cause preserved and the dataset URI in the message. Previously, Lance JNI/runtime exceptions escaped raw, so the tiering orchestrator could not treat them uniformly with other IO failures and operators had no context about which dataset failed.close()now catchesIllegalStateExceptionthat Arrow'sRootAllocatorraises when child buffers were leaked, and rethrows it as anIOExceptionwith an actionable message identifying the source. The allocator is still closed on happy paths and itsclose()call is no longer skipped by upstream failures - the exact leak text is kept as the cause so that debugging remains straightforward.Add
LanceLakeCommitterTestcovering:close()after construction with no work performed does not leak.commit()against a non-existent dataset raisesIOExceptionwith aRuntimeExceptioncause.getMissingLakeSnapshot()against a missing dataset can be called and the committer can still be closed cleanly afterwards.Purpose
Improve the failure-side robustness of the Lance tiering committer so that Lance-specific runtime errors are surfaced uniformly as
IOExceptions (matching theLakeCommittercontract), the underlying Arrow allocator is always released on close, and any leaked child buffers produce an actionable error message that clearly identifies the source. This unblocks reliable retry / abort handling by the tiering orchestrator and reduces the operational cost of diagnosing tiering failures against Lance datasets.Linked issue: close #xxx
Brief change log
LanceLakeCommitter#commit: wrapRuntimeExceptionfromLanceDatasetAdapter.commitAppendasIOException, preserving the original cause and including the dataset URI in the message.LanceLakeCommitter#close: guardallocator.close()with a try/catch that converts Arrow'sIllegalStateException(raised on leaked child buffers) into anIOExceptionwith an actionable message, keeping the original stack as the cause.LanceLakeCommitterTestunderfluss-lake/fluss-lake-lance/src/test/java/org/apache/fluss/lake/lance/tiering/covering the three failure-side branches described above.Tests
LanceLakeCommitterTest(3 test methods) covers:testCloseReleasesAllocatorEvenIfCommitNeverCalledtestCommitOnMissingDatasetIsWrappedAsIOExceptiontestGetMissingLakeSnapshotReturnsNullWhenDatasetAbsent./mvnw -pl fluss-lake/fluss-lake-lance test -Dspotless.check.skip— 22/22 pass, no regressions in existingLanceTieringTest/LanceArrowUtilsTest/ArrowDataConverterTest../mvnw -pl fluss-lake/fluss-lake-lance checkstyle:check— 0 violations.API and Format
LanceLakeCommitter#commitcontinues to declarethrows IOException; only the exception type produced on the failure path is tightened (previouslyRuntimeExceptioncould escape). Callers that already handledIOExceptionfromLakeCommitter#commitnow get a strictly better contract.LanceLakeCommitter#closecontinues to declarethrows Exception; leaks now surface asIOExceptionwith a clear message instead of rawIllegalStateExceptionfrom Arrow.Documentation
LanceLakeCommitterso that future contributors do not accidentally revert the wrapping.