Skip to content

feat(scan): [3/N] read and apply V3 deletion vectors - #3035

Draft
mbutrovich wants to merge 14 commits into
apache:mainfrom
mbutrovich:dv-apply
Draft

feat(scan): [3/N] read and apply V3 deletion vectors#3035
mbutrovich wants to merge 14 commits into
apache:mainfrom
mbutrovich:dv-apply

Conversation

@mbutrovich

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

Draft PR, stacked on #2868. This is Task 3 of the deletion-vector read epic: reading and applying V3 deletion vectors during a scan.

  • CachingDeleteFileLoader reads a deletion vector's deletion-vector-v1 blob directly from its Puffin file by byte range (content_offset / content_size_in_bytes), decrypting first via EncryptedInputFile when the entry carries key metadata.
  • After decoding, the blob's cardinality is checked against the manifest entry's record_count, matching the same check Iceberg-Java does in BitmapPositionDeleteIndex.deserializeBitmap. Mismatch is a DataInvalid error, not a silent wrong result.
  • DeleteFileIndex now indexes deletion vectors by referenced_data_file and applies them ahead of position delete files, since a DV supersedes any position deletes for the same data file per spec.
  • A few spec invariants are now enforced as errors instead of silently mishandled:
    • a PositionDeletes entry with content_offset set but no referenced_data_file
    • two deletion vectors referencing the same data file (at most one is allowed per data file per snapshot)
    • a deletion vector whose partition or sequence number disagrees with the data file it claims to reference
  • FileScanTaskDeleteFile gained a record_count field, populated from the manifest entry, used for the cardinality check above.
  • DeleteVector::deserialize (from feat(delete-vector): [1/N] decode deletion-vector-v1 puffin blobs #2866) is now actually wired in; removed its #[allow(dead_code)].

No data-file read path changes outside of deletion vectors. Existing position delete and equality delete handling is unchanged.

Are these changes tested?

Yes.

  • Unit tests in delete_file_index.rs for: DV supersedes partition-scoped and path-scoped position deletes, DV coexists with equality deletes, rejecting a DV missing referenced_data_file, rejecting duplicate DVs for one data file, rejecting a DV with a mismatched partition or stale sequence number.
  • Unit tests in caching_delete_file_loader.rs for: reading and applying a DV, reading an encrypted DV, rejecting a cardinality mismatch, and the coordinate/cardinality validation helpers directly.
  • End-to-end tests in positional_deletes.rs reading a real Parquet data file with a DV applied through ArrowReader, including the cardinality-mismatch failure path through the full read.

AI Disclosure

Developed with the help of Claude Code, but I understand and support these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[EPIC] Deletion vector read support (deletion-vector-v1 Puffin blobs)

1 participant