Skip to content

[Bug] Validate deletion-vector checksums when reading #9992

Description

@Akash3121

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

master (1d368b4)

Compute Engine

Engine-independent core storage. Reproduced as a unit test against DeletionVector.read.

Minimal reproduce step

  1. Create and serialize a BitmapDeletionVector or Bitmap64DeletionVector.
  2. Change only the final checksum byte.
  3. Read the bytes through DeletionVector.read.

The corrupted vector is accepted.

Both writers append a CRC32 checksum:

  •  BitmapDeletionVector#serializeTo 
  •  Bitmap64DeletionVector#serializeTo 

The reader does not read or compare that checksum. Both branches currently execute:

 dis.skipBytes(4); // skip crc

 skipBytes(4)  also does not guarantee that four bytes were available, so a truncated checksum can be accepted.

What doesn't meet your expectations?

A mismatched or truncated deletion-vector checksum should fail before the bitmap is used.

Silently accepting corruption can cause deleted rows to reappear or live rows to disappear when corrupted bitmap data remains structurally parseable.

Anything else?

The reader should read the stored checksum, calculate the CRC over the same byte range used by the writer, compare the values, and explicitly reject missing checksum bytes or invalid lengths.

Suggested tests for both 32-bit and 64-bit encodings:

  1. Corrupted checksum
  2. Corrupted payload
  3. Truncated checksum
  4. Corruption read through a real deletion-vector index file and offset
  5. Compatibility with existing valid fixtures

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions