feat: add Comet CachedBatchSerializer for native in-memory cache - #4569

Closed

andygrove wants to merge 20 commits into

apache:mainfrom

andygrove:comet-cached-batch-serializer

Commits on Jun 2, 2026

feat: add spark.comet.cache.serializer.enabled config
andygrove
committed
27e07a7
feat: add CometCachedBatch and column stats helper
andygrove
committed
0a9c233
fix: use ByteArray.compareBinary for version-safe string stats ordering
andygrove
committed
82517f7
test: exercise CometCachedBatch and document setRowCount ordering
andygrove
committed
cf3a73c
feat: add CometCachedBatchSerializer skeleton with schema routing
andygrove
committed
6c80ef0
refactor: simplify CometCachedBatchSerializer schema routing and imports
andygrove
committed
ec936f9
feat: encode cached batches as compressed Arrow IPC with stats
andygrove
committed
fe21619
feat: support TimestampNTZ in cache stats and harden build test
andygrove
committed
bf522db
feat: decode Comet cached batches with column pruning and row fallback
andygrove
committed
29cb511
test: cover columnar cache read path and clarify selectedIndices errors
andygrove
committed
4f4aa16
feat: pass already-Arrow batches through CometSparkToColumnarExec without copy
andygrove
committed
3c1cd5f
refactor: import CometVector and clarify passthrough test/metric
andygrove
committed
826249f
feat: install Comet cache serializer from CometDriverPlugin when enabled
andygrove
committed
4960cfb
refactor: use StaticSQLConf.SPARK_CACHE_SERIALIZER and log serializer install
andygrove
committed
bec8c30
test: end-to-end Comet cache serializer correctness, pruning, spill, delegation, ntz
andygrove
committed
7ba47f4
test: assert cached values (not just counts) for filter and timestamp_ntz
andygrove
committed
9dc0f7d
fix: copy UTF8String for cache stats to avoid use-after-free on string bounds
andygrove
committed
40184ce
chore: open as draft pull request [skip ci]
andygrove
committed
d1af3a3
refactor: dedup cache decode path and precompute stat orderedness
andygrove
committed
bedaa77
chore: keep draft CI skipped after cleanup [skip ci]
andygrove
committed
920c57f

Navigation Menu

feat: add Comet CachedBatchSerializer for native in-memory cache - #4569

feat: add Comet CachedBatchSerializer for native in-memory cache#4569
andygrove wants to merge 20 commits into
apache:mainfrom
andygrove:comet-cached-batch-serializer

Commits on Jun 2, 2026

feat: add spark.comet.cache.serializer.enabled config

feat: add CometCachedBatch and column stats helper

fix: use ByteArray.compareBinary for version-safe string stats ordering

test: exercise CometCachedBatch and document setRowCount ordering

feat: add CometCachedBatchSerializer skeleton with schema routing

refactor: simplify CometCachedBatchSerializer schema routing and imports

feat: encode cached batches as compressed Arrow IPC with stats

feat: support TimestampNTZ in cache stats and harden build test

feat: decode Comet cached batches with column pruning and row fallback

test: cover columnar cache read path and clarify selectedIndices errors

feat: pass already-Arrow batches through CometSparkToColumnarExec without copy

refactor: import CometVector and clarify passthrough test/metric

feat: install Comet cache serializer from CometDriverPlugin when enabled

refactor: use StaticSQLConf.SPARK_CACHE_SERIALIZER and log serializer install

test: end-to-end Comet cache serializer correctness, pruning, spill, delegation, ntz

test: assert cached values (not just counts) for filter and timestamp_ntz

fix: copy UTF8String for cache stats to avoid use-after-free on string bounds

chore: open as draft pull request [skip ci]

refactor: dedup cache decode path and precompute stat orderedness

chore: keep draft CI skipped after cleanup [skip ci]