[core] Rebuild file index over the correct column after schema evolution - #10122
Conversation
FileIndexProcessor computes projectedIndexCols as the index columns' positions in the file schema, and creates the index writer with fileSchema.project(projectedIndexCols). The reader, however, was built with ReadBuilder.withProjection(projectedIndexCols), which resolves the indices against the current table schema. Once a schema change shifts column positions (for example dropping a middle column and adding another), the file-schema index no longer matches the table-schema index, so the reader returns a different column and the file index is rebuilt over it. A later query on the indexed column then silently prunes files that actually match. Read the columns with withReadType(fileSchema.project(projectedIndexCols)), the same file-schema projection the writer uses, so the reader and writer stay in the file's own column space. rewrite_file_index reads a single data file whose committed schema is that file schema, and file indexes are probed at query time with predicates devolved to the file schema, so the per-file index must be built in the file's column and type space. The name-mapping half of schema evolution was added with the rewrite_file_index procedure in apache#6562, but the reader projection was never aligned to it.
| Map<String, String> options = new HashMap<>(); | ||
| options.put(CoreOptions.BUCKET.key(), "1"); | ||
| options.put(CoreOptions.FILE_FORMAT.key(), "parquet"); | ||
| options.put(CoreOptions.FILE_INDEX + ".bloom-filter.columns", "v"); |
There was a problem hiding this comment.
non blocking
Could this test use file-index.bitmap.columns instead of a Bloom filter? A Bloom-filter positive is probabilistic, so remain() == true does not strictly prove that 100 was indexed from v ; a false positive could allow the old wrong-column behavior to pass. The bitmap index returns an exact empty result for an absent value, making this regression deterministic while preserving the same schema-evolution scenario.
Switch the schema-evolution regression test from a bloom filter to a bitmap index. A bloom-filter positive is probabilistic, so remain() == true does not strictly prove the value was indexed from the correct column - a false positive could let the old wrong-column behavior pass. A bitmap index returns an exact empty result for an absent value, so the test now reads the (embedded) index and asserts the indexed value is present while an absent value is not.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. The rewrite_file_index path can rebuild an index from the wrong column when a data file's schema predates a drop/add that shifts column positions. That can make subsequent index pruning return incorrect results. This change aligns the reader's file-schema projection with the index writer's projection.
Implementation: CLEAN. I traced FileIndexProcessor through ReadBuilderImpl.withProjection/withReadType and the file-index serialization path. The new regression uses a real table write and schema evolution, then inspects an exact bitmap index, so it tests the failure mode without a probabilistic bloom-filter assertion. I found no actionable issue in the current diff.
Verification on this head (c6e4764), JDK 8: mvn -pl paimon-core -am -Pfast-build -DskipTests package succeeded; then mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=FileIndexProcessorTest test passed (3 tests, 0 failures/errors). The required CI checks on this head are green. I did not run a full Flink or Spark procedure job locally.
apache#10122 fixed the rewrite_file_index read-projection drift after schema evolution and shipped a test for the same-arity column-shift case. Add the arity-shrink variant it did not cover: dropping a column before the indexed one leaves a file-schema position past the current schema arity, which was the IndexOutOfBoundsException path.
3 participants
Purpose
FileIndexProcessor(therewrite_file_indexprocedure) rebuilds a data file's file index. It computesprojectedIndexColsas the index columns' positions in the file schema and builds the writer withfileSchema.project(projectedIndexCols). But the reader usedReadBuilder.withProjection(projectedIndexCols), whichReadBuilderImplresolves against the current table schema.For a file whose
schemaIddiffers from current, the spaces diverge once columns shift:[k, a, v](bloom onv) → dropa, addwleavesvat file-index 2 while current-index 2 is noww. The reader then readsw(null-filled for the old file), so the bloom is rebuilt over nulls and a laterv = 100query silently prunes the file.Fix: read with
withReadType(fileSchema.project(projectedIndexCols))— the writer's own file-schema projection — so reader and writer share the file's column space. Current-schema files are unchanged.Tests
testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd: bloom onvin[k, a, v], writev=100, dropa, addw, rebuild, assert thevbloom still reports100present. Fails on master (wrong column →100absent); passes here.