[format][python] Coalesce map blob metadata reads - #10121
Merged
JingsongLi merged 3 commits intoSep 23, 2026
Merged
JingsongLi merged 3 commits into
JingsongLi merged 3 commits into
Conversation
Akash3121
approved these changes
Sep 23, 2026
| List<long[]> ranges = trackingIO.lastInputStream.readRanges; | ||
| if (entryCount == 32) { | ||
| // File footer/index plus map header, lengths, combined indexes and keys. | ||
| assertThat(ranges).hasSize(6); |
Contributor
There was a problem hiding this comment.
Nit: avoid coupling this regression to the readers total req count
This exact count also includes unrelated file-footer and row-index reads, so a future optimization or refactor in those readers could break this test even if the map metadata remains correctly coalesced. Could we assert the relevant map index/key ranges directly, or use an upper bound here as in the large-metadata case? That would preserve the optimization guarantee without coupling the test to the complete reader request sequence.
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Coalesce adjacent
MAP<X, BLOB>indexes in Python and Java. Java also buffers consecutive keys with bounded read-ahead, avoiding per-key reads without touching BLOB values.Detect native commit APIs so older Rust runtimes retain Python fallback.
Tests