Search before asking
Paimon version
master (1d368b4)
Compute Engine
Engine-independent core storage. Reproduced as a unit test against DeletionVector.read.
Minimal reproduce step
- Create and serialize a BitmapDeletionVector or Bitmap64DeletionVector.
- Change only the final checksum byte.
- Read the bytes through DeletionVector.read.
The corrupted vector is accepted.
Both writers append a CRC32 checksum:
- BitmapDeletionVector#serializeTo
- Bitmap64DeletionVector#serializeTo
The reader does not read or compare that checksum. Both branches currently execute:
dis.skipBytes(4); // skip crc
skipBytes(4) also does not guarantee that four bytes were available, so a truncated checksum can be accepted.
What doesn't meet your expectations?
A mismatched or truncated deletion-vector checksum should fail before the bitmap is used.
Silently accepting corruption can cause deleted rows to reappear or live rows to disappear when corrupted bitmap data remains structurally parseable.
Anything else?
The reader should read the stored checksum, calculate the CRC over the same byte range used by the writer, compare the values, and explicitly reject missing checksum bytes or invalid lengths.
Suggested tests for both 32-bit and 64-bit encodings:
- Corrupted checksum
- Corrupted payload
- Truncated checksum
- Corruption read through a real deletion-vector index file and offset
- Compatibility with existing valid fixtures
Are you willing to submit a PR?
Search before asking
Paimon version
master (1d368b4)
Compute Engine
Engine-independent core storage. Reproduced as a unit test against DeletionVector.read.
Minimal reproduce step
The corrupted vector is accepted.
Both writers append a CRC32 checksum:
The reader does not read or compare that checksum. Both branches currently execute:
skipBytes(4)also does not guarantee that four bytes were available, so a truncated checksum can be accepted.What doesn't meet your expectations?
A mismatched or truncated deletion-vector checksum should fail before the bitmap is used.
Silently accepting corruption can cause deleted rows to reappear or live rows to disappear when corrupted bitmap data remains structurally parseable.
Anything else?
The reader should read the stored checksum, calculate the CRC over the same byte range used by the writer, compare the values, and explicitly reject missing checksum bytes or invalid lengths.
Suggested tests for both 32-bit and 64-bit encodings:
Are you willing to submit a PR?