[Bug] (bitmap) Support SET type (Flag=5) deserialization in FE BitmapValue

[iyils]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master, commit_id:6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907

What's Wrong?

FE BitmapValue.java does not handle BitmapTypeCode.SET (Flag = 5) in its deserialize() method. When BE has enable_set_in_bitmap_value=true, or after #65421 merges wider SET adoption, the FE throws RuntimeException: unknown bitmap type 5 when reading such bitmaps. This causes failures in Spark Load, Hive Catalog bitmap import (bitmap_from_base64), and any path that routes a BE-serialized bitmap through FE.

What You Expected?

FE should be able to deserialize all valid bitmap binary formats produced by BE,
including SET type (Flag=5), without throwing an exception.

The deserialized bitmap should contain the same set of values as the original bitmap
serialized by BE.

How to Reproduce?

Prerequisites:

• Apache Doris master branch (enable_set_in_bitmap_value=true by default since [Chore](bitmap) set enable_set_in_bitmap_value default value to true #35730)

Step 1: Export a SET-format bitmap from BE

On BE, a bitmap with 1–32 elements is serialized as SET type (Flag=5).
Use bitmap_to_base64 to export the binary:

SELECT bitmap_to_base64(bitmap_from_array([1, 3, 5]));
-- Result: BQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA=

The decoded bytes start with 0x05 (Flag=5 = SET type).

Step 2: Try to parse the exported binary in FE
The current BitmapValue.java#deserialize() (line 131–154) only handles EMPTY / SINGLE32 / SINGLE64 / BITMAP32 / BITMAP64, and throws on anything else:

// fe/fe-common/src/main/java/org/apache/doris/common/io/BitmapValue.java L131-154
public void deserialize(DataInput input) throws IOException {
    clear();
    int bitmapType = input.readByte();
    switch (bitmapType) {
        case EMPTY:
            break;
        case SINGLE32:
            singleValue = Util.toUnsignedLong(Integer.reverseBytes(input.readInt()));
            this.bitmapType = SINGLE_VALUE;
            break;
        case SINGLE64:
            singleValue = Long.reverseBytes(input.readLong());
            this.bitmapType = SINGLE_VALUE;
            break;
        case BITMAP32:
        case BITMAP64:
            bitmap = bitmap == null ? new Roaring64Map() : bitmap;
            bitmap.deserialize(input, bitmapType);
            this.bitmapType = BITMAP_VALUE;
            break;
        default:
            throw new RuntimeException(String.format("unknown bitmap type %s ", bitmapType));
    }
}

Feeding the base64 output from Step 1 into this method triggers the default branch:

byte[] bytes = Base64.getDecoder().decode("BQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA=");
DataInput input = new DataInputStream(new ByteArrayInputStream(bytes));

BitmapValue bitmap = new BitmapValue();
bitmap.deserialize(input);  // throws RuntimeException: unknown bitmap type 5

Root Cause:
deserialize() has no case for SET (Flag=5) or SET_V2 (Flag=10). Since BE defaults enable_set_in_bitmap_value=true (since #35730), any bitmap with ≤32 elements is serialized as SET, making this a regression on all FE read paths (Hive Catalog, Spark Load, etc.).

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial analysis

Verdict: confirmed Java-reader compatibility gap at the reported commit, with corrections needed to the reproduction and stated blast radius. No Doris profile or runtime logs are needed to establish the class-level defect.

Verified facts

• At 6b3ea1eb56dcf57c5f2dd5abb9f170e0fbcf1907, BE defaults enable_set_in_bitmap_value to true (config). Its binary contract defines SET = 5, and the SET writer emits type:u8, count:u8, then count little-endian uint64 values (type codes, writer). The inline SET capacity is 32 (definition). BE's own reader already handles flag 5 and rejects counts above 32 (reader).
• FE BitmapValue defines only codes 0 through 4 and its deserialize() switch handles only those codes; flag 5 deterministically reaches the reported unknown bitmap type 5 default branch (FE constants and reader). The reader switch is at lines 131-153.
• Existing FE coverage is a Java-writer-to-Java-reader round trip. Since the Java writer cannot emit SET, it does not test BE/FE wire compatibility (test).

Reproduction correction

The Base64 value in the issue, BQMFAAAAAAAAAEAAAAAAAAAAwAAAAAAAAA=, is malformed (35 characters). Base64.getDecoder().decode(...) throws IllegalArgumentException: Input byte array has wrong 4-byte ending unit, so the posted Java snippet does not reach BitmapValue.deserialize() as written.

A valid flag-5 fixture containing {1, 3, 5} is:

BQMBAAAAAAAAAAMAAAAAAAAABQAAAAAAAAA=

It decodes to 26 bytes: type 5, count 3, followed by little-endian 1, 3, and 5. Passing those bytes to the cited FE reader reaches unknown bitmap type 5. SET payload order is not part of the semantic contract, so another ordering of the same three values is also valid if correctly encoded.

Impact qualification / missing evidence

• The Java Hive UDF module definitely uses this reader for binary bitmap arguments (utility), so a Hive UDF consuming BE-produced SET bytes is a verified affected boundary.
• Doris SQL bitmap_from_base64 itself is implemented and deserialized in BE, whose reader supports SET (implementation). Therefore that builtin alone is not evidence of an FE Java failure. Please provide the full stack trace and the exact Hive Catalog/import call path if a catalog workflow routes the bytes through fe-common first.
• The Spark Load impact is plausible from the purpose of this legacy Java bitmap class, but the issue does not include a failing job or stack trace. Please add the exact load mode/artifact version and full exception stack if this broader impact needs to be tracked as verified rather than inferred.

These missing call-path details do not block fixing the deterministic flag-5 reader defect.

Recommended next steps

1. Add a SET = 5 read case in FE: read the count as an unsigned byte, enforce the 32-element format limit, read each value as little-endian 64-bit, and populate the existing Java single/bitmap representation. A distinct in-memory SET representation is unnecessary for semantic compatibility.
2. Add hard-coded cross-language fixtures rather than another Java self-round-trip. Cover a normal SET, the 32-element boundary, a 64-bit value, truncated input, count greater than 32, and the chosen duplicate-value policy (BE rejects duplicate values in a flag-5 payload).
3. Clarify the PR scope relative to the expectation of “all valid bitmap binary formats.” The Java reader also lacks BE's SET_V2 = 10, BITMAP32_V2 = 12, and BITMAP64_V2 = 13 codes (BE type table). BE defaults bitmap_serialize_version to 1, so flag 5 is the immediate default-path bug, but adding only flag 5 should not be described as complete support for every BE format.

Breakwater-GitHub-Analysis-Slot: slot_595e6f9856b0

[eveniota]

I would like to work on this issue.
I will add SET (Flag=5) case to deserialize() with:

• count validation (≤32)
• Little-endian uint64 reading
• Cross-language test fixtures

edit: Sorry i didn't noticed that you are willing to submit a PR. Go on i am not working on this issue.