Skip to content

[core] Preserve sequence-group deletes in projected reads - #10135

Merged
JingsongLi merged 1 commit into
apache:masterfrom
taoran92:fix/partial-update-sequence-group-projection
Sep 24, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
taoran92:fix/partial-update-sequence-group-projection

Conversation

@taoran92

@taoran92 taoran92 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #10134.

Column pruning can drop sequence fields needed to determine whether a partial-update row has been deleted. Consequently, COUNT(*) and projected reads can return deleted rows even when SELECT * correctly excludes them.

Include whole-row deletion fields when deriving internal read dependencies and expand their complete ordering fields, including sequence groups with multiple ordering fields. Preserve the user-visible projection.

Tests

  • PartialUpdateMergeFunctionTest: 47 tests passed, covering empty, key-only, unrelated-field and reordered projections; single and multiple ordering fields; and newer, equal, older and null delete sequences.
  • Spark 3.5.8 DeleteFromTableTest and V2DeleteFromTableTest: all 64 tests passed. The new regression uses two independent field-group writes and an explicit delete record, then checks full-row, count, key-only and other-group reads.

@taoran92
taoran92 force-pushed the fix/partial-update-sequence-group-projection branch from 1ebe383 to 284eb9e Compare September 23, 2026 09:04
@JingsongLi

Copy link
Copy Markdown
Contributor

This fixes a correctness issue with direct user impact: projected reads and COUNT(*) can otherwise retain rows deleted through a sequence group. I traced adjustReadType through create(readType): adding the delete group’s protected field to the required set causes its full comparator ordering fields to be read, while the requested projection remains the visible schema. The new tests cover empty, reordered, and other-group projections plus composite ordering.

Local verification on the isolated patch: all 47 PartialUpdateMergeFunctionTest cases passed on JDK 8. I also built the Spark 3.5 test module, but its suite aborted before executing tests because this sandbox blocks Spark’s local socket bind (java.net.SocketException: Operation not permitted). The current PR checks show failures in Spark 3 and Flink 1 Connectors/CDC, so please resolve or clearly establish those failures as unrelated before merge; I cannot call this production-ready while CI is red. The fix itself has strong end-to-end value.

Retain all ordering fields required by sequence groups that can delete
whole rows, even when those fields are absent from the query projection.

Add tests for empty, key-only and unrelated-field projections,
including sequence groups with multiple ordering fields.
@taoran92
taoran92 force-pushed the fix/partial-update-sequence-group-projection branch from 284eb9e to 282667b Compare September 24, 2026 02:30
@JingsongLi
JingsongLi merged commit dfa2c0c into apache:master Sep 24, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Projected reads return deleted rows for partial-update tables with sequence-group deletion

2 participants