[core] Validate chain partition drops against the post-commit state - #10127
JingsongLi merged 3 commits into
Conversation
ChainTableCommitPreCallback rejects dropping a snapshot partition whose delta followers would lose their baseline. The predecessor and successor candidates came from the pre-commit partition list, so when one commit dropped several partitions of a group at once (batch INSERT OVERWRITE or a rollback), a partition dropped by that same commit still counted as the predecessor of the next one: the check passed, the commit landed, and the delta partitions silently fell back to delta-only reads without their baseline rows. Validate the post-commit state instead. A partition counts as dropped only when the commit deletes all of its base files; a rollback deletes per-file and may leave a partition partially alive, and such a partition survives and still anchors its delta followers, so it is neither excluded from the candidates nor itself re-validated. Only fully dropped partitions are validated, and single-partition drops are unaffected.
…artitions The post-commit-state validation rejected a legitimate batch INSERT OVERWRITE. ChainTableOverwriteCommitCallback truncates the snapshot partitions that the overwrite just rewrote on the delta branch, and the pre-callback saw the freshly rewritten delta followers as stranded and threw "Snapshot partition cannot be dropped" (SparkChainTableITCase). Those followers hold fresh, complete data and need no baseline. Pass the freshly rewritten delta partitions from the overwrite callback to the pre-callback via a thread-local scoped around the synchronous truncate, and exclude them from the follower check. A standalone drop or rollback rewrites no delta, so a genuinely stranded follower is still rejected.
Follow-up from reviewing the freshly-written-partition handoff. Save and restore the previous scope value instead of unconditionally clearing it, so a future nested chain overwrite on the same thread cannot wipe an outer scope, and return an unmodifiable view of the set from the getter.
|
This closes a real data-integrity gap: a same-commit-dropped snapshot partition must not act as the surviving baseline for a delta follower. I traced the regular truncate path and rollback path into the pre-callback: regular OVERWRITE commits request conflict detection over changed partitions, and rollback supplies the latest snapshot’s base entries; the new fully-dropped check distinguishes whole-partition loss from a partial file delete. The overwrite scope is restored in Local verification on the PR patch: all 17 One scale concern for production: |
|
Thank you @JingsongLi |
2 participants
Purpose
close #10126
ChainTableCommitPreCallbackrejects dropping a snapshot partition of a chain table whose delta followers would then lose their baseline (they would silently fall back to delta-only reads). Its predecessor candidates came from the pre-commit partition list, so when one commit dropped several partitions of a chain group at once (a batch INSERT OVERWRITE or a rollback), a partition dropped by that same commit still counted as the predecessor of the next one. The check passed, the commit landed, and the delta partitions lost their baseline rows.This validates the post-commit state instead.
fullyDroppedPartitionstreats a partition as dropped only when the commit deletes all of its base files; a rollback deletes per-file and may leave a partition partially alive, and such a partition survives and still anchors its delta followers. Only fully dropped partitions are excluded from the candidate list and validated, so a same-commit-dropped partition can no longer masquerade as another's predecessor. Standalone drops are unaffected.A chain-table OVERWRITE legitimately truncates a whole group's snapshot partitions:
ChainTableOverwriteCommitCallbacktruncates the snapshot partitions that the overwrite just rewrote on the delta branch, and those partitions' surviving delta followers hold fresh, complete data that needs no baseline. The overwrite callback hands the freshly rewritten delta partitions to the pre-callback through a thread-local scoped around its synchronous truncate, and the pre-callback excludes them from the follower check. A standalone drop or rollback rewrites no delta, so a genuinely stranded follower is still rejected.Tests
ChainTablePartitionExpireTest#testRollbackRejectedWhenBatchDroppingBaselinesOfDelta: one rollback drops bothCN/20250201andCN/20250301of a chain group while a deltaCN/20250315follows. Pre-fix the pre-commit list letCN/20250201serve asCN/20250301's predecessor, so the drop was accepted and the delta lost its baseline; the test asserts the rollback is now rejected and the latest snapshot is unchanged.ChainTablePartitionExpireTest#testRollbackAllowedWhenPartitionOnlyPartiallyDeleted: a rollback that deletes only one of two files ofCN/20250201, leaving it alive, while fully droppingCN/20250301, is allowed because the survivingCN/20250201still anchors the delta. This guards against a false rejection on a partial delete.SparkChainTableITCase#testChainTableWithMultiGroupPartition(existing) covers the whole-group INSERT OVERWRITE path that the freshly-rewritten-partition exclusion keeps working.API and Format
no
Documentation
no