Skip to content

[format][python] Coalesce map blob metadata reads - #10121

Merged
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/coalesce-python-blob-map-indexes
Sep 23, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/coalesce-python-blob-map-indexes

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Coalesce adjacent MAP<X, BLOB> indexes in Python and Java. Java also buffers consecutive keys with bounded read-ahead, avoiding per-key reads without touching BLOB values.

Detect native commit APIs so older Rust runtimes retain Python fallback.

Tests

  • Commit/BLOB tests: 242 passed with the development Rust runtime; 204 passed and 38 skipped with Rust 0.3.0.
  • 170 Python BLOB tests and 44 Java BLOB tests passed; Flake8 and full Maven checks passed.
  • Range-tracking tests cover descriptor correctness, null/empty values, buffer boundaries, and no value-data reads.
  • Real OSS sample with 17 entries: Java metadata Range GETs 21 → 4, unchanged 854 bytes and descriptor checksum; median 0.931s → 0.217s over three runs per version. Measured with a direct Range-GET stream, excluding catalog/scan time.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Coalesce map blob index reads [format][python] Coalesce map blob metadata reads Sep 22, 2026
List<long[]> ranges = trackingIO.lastInputStream.readRanges;
if (entryCount == 32) {
// File footer/index plus map header, lengths, combined indexes and keys.
assertThat(ranges).hasSize(6);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: avoid coupling this regression to the readers total req count
This exact count also includes unrelated file-footer and row-index reads, so a future optimization or refactor in those readers could break this test even if the map metadata remains correctly coalesced. Could we assert the relevant map index/key ranges directly, or use an upper bound here as in the large-metadata case? That would preserve the optimization guarantee without coupling the test to the complete reader request sequence.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit c29ddcb into apache:master Sep 23, 2026
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants