Core, Parquet: Carry avg value sizes for v4 content stats - #17451
Open
huan233usc wants to merge 1 commit into
Open
Core, Parquet: Carry avg value sizes for v4 content stats#17451huan233usc wants to merge 1 commit into
huan233usc wants to merge 1 commit into
Conversation
github-actions
Bot
added
API
spark
parquet
core
Specification
huan233usc
marked this pull request as draft
July 31, 2026 14:54
huan233usc
force-pushed
the
geo-avg-value-size-manifest
branch
from
July 31, 2026 17:45
018c802 to
4bdbfaf
Compare
huan233usc
changed the title
huan233usc
force-pushed
the
geo-avg-value-size-manifest
branch
from
August 19, 2026 18:32
4bdbfaf to
0b172a2
Compare
huan233usc
changed the title
huan233usc
force-pushed
the
geo-avg-value-size-manifest
branch
2 times, most recently
from
August 20, 2026 02:09
adf31b5 to
8e3e2a1
Compare
Propagate average non-null value sizes from Parquet metrics through Metrics and ContentFile so v4 content stats adapters can preserve them. The v1-v3 manifest schemas are unchanged, so v3 manifests do not persist this optional metric. No cross-version Java-serialization shim is added: no type holds a Metrics field, so Metrics is only ever serialized and deserialized within a single version.
huan233usc
force-pushed
the
geo-avg-value-size-manifest
branch
from
August 20, 2026 04:10
8e3e2a1 to
f139886
Compare
huan233usc
marked this pull request as ready for review
August 20, 2026 04:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #17333.
#17333 collects the average serialized WKB size in
FieldMetrics, butParquetMetricsdiscarded it while assemblingMetrics, so the value could never reach aDataFileor the v4content_statsadapters. This PR closes that gap, carrying per-field average non-null value sizes throughMetrics,ContentFile, the file builders, copies, and filtering, and exposing them through the legacyContentFileview that v4 manifest readers use.The v1-v3 manifest schemas are unchanged, so v3 manifests do not persist this optional metric. The v4 write-direction wrapper in #16936 can consume
ContentFile.avgValueSizes()after rebasing.ContentFile.avgValueSizes()is adefaultmethod returning null. Adding an abstract method toContentFilewould break the public API and ABI (RevAPI reportsjava.method.addedToInterface), and implementations that carry no column stats, such as the deletion-vector adapter, correctly inherit null.Metricsalso keeps its unpinnedserialVersionUID: no type holds aMetricsfield, so it is only ever serialized and deserialized within a single version, and no cross-version shim is needed.Tests:
./gradlew :iceberg-api:test --tests org.apache.iceberg.TestMetricsSerialization./gradlew :iceberg-core:test --tests org.apache.iceberg.TestContentStatsBackedMap --tests org.apache.iceberg.TestTrackedFileAdapters./gradlew :iceberg-data:test --tests org.apache.iceberg.parquet.TestParquetMetrics.testMetricsForGeospatialTypes./gradlew :iceberg-parquet:test --tests org.apache.iceberg.parquet.TestParquetDataWriter.testGeospatialRoundTrip./gradlew :iceberg-api:revapi :iceberg-core:revapi :iceberg-parquet:revapi