[core] Add regression test for rewrite_file_index over an arity-shrinking column drop - #10092
LuciferYang wants to merge 1 commit into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The linked issue is a production correctness bug: after a column drop, rewrite_file_index can fail on old files or silently build an index under one column's name from another column's values. This change resolves the read projection in the current table schema while preserving the file-schema projection for the index writer, keeping their row positions aligned by field name. I checked dropped and re-added columns, current-schema consistency under concurrent DDL, and nested map-column de-duplication; no actionable regression was found.
The two new tests cover both the out-of-range failure and same-arity silent-corruption case, asserting actual bloom-filter values. They could not reach the rewritten path in my local Maven run because the test fixture failed while loading the existing CodeGenerator service (the same local failure occurs in unrelated core tests). The PR head's JDK 8/11, Flink 1/2, Spark, and E2E CI checks all pass.
|
Please rebase master. |
apache#10122 fixed the rewrite_file_index read-projection drift after schema evolution and shipped a test for the same-arity column-shift case. Add the arity-shrink variant it did not cover: dropping a column before the indexed one leaves a file-schema position past the current schema arity, which was the IndexOutOfBoundsException path.
8a770d0 to
8c972df
Compare
|
Repurposed to test-only. #10122 landed the same fix (reading the rewrite projection through the file schema), so I dropped the now-redundant production change and the test that duplicated its |
Purpose
The
rewrite_file_indexread-projection drift after schema evolution (#10091) is already fixed on master by #10122, which reads the rewrite projection through the file schema. That fix shipped one regression test,testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd, for the same-arity case: aDROP COLUMNplus a laterADD COLUMNthat shifts the indexed column onto a different current position.This PR adds the variant the same fix already covers but that no test pins: a
DROP COLUMNbefore the indexed column, which shrinks the schema arity. There a file-schema position of the indexed column can exceed the current schema arity, so it takes the out-of-range path (anIndexOutOfBoundsExceptionbefore #10122) rather than the same-arity wrong-column path.There is no production change here; this is test-only coverage on top of #10122.
Tests
FileIndexProcessorTest#testProcessAfterDroppingColumnBeforeIndexedColumn: a file is written at schema[k, a, v]with a bloom filter onv, thenais dropped so the current schema is[k, v]while the file keepsvat file position 2. The test rewrites the file index and asserts a side index file is written and holds thevindex. Against the pre-#10122 code the rewrite failed here withIndexOutOfBoundsException.API and Format
no
Documentation
no