Skip to content

[core] Add regression test for rewrite_file_index over an arity-shrinking column drop - #10092

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/file-index-rewrite-projection
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/file-index-rewrite-projection

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

The rewrite_file_index read-projection drift after schema evolution (#10091) is already fixed on master by #10122, which reads the rewrite projection through the file schema. That fix shipped one regression test, testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd, for the same-arity case: a DROP COLUMN plus a later ADD COLUMN that shifts the indexed column onto a different current position.

This PR adds the variant the same fix already covers but that no test pins: a DROP COLUMN before the indexed column, which shrinks the schema arity. There a file-schema position of the indexed column can exceed the current schema arity, so it takes the out-of-range path (an IndexOutOfBoundsException before #10122) rather than the same-arity wrong-column path.

There is no production change here; this is test-only coverage on top of #10122.

Tests

FileIndexProcessorTest#testProcessAfterDroppingColumnBeforeIndexedColumn: a file is written at schema [k, a, v] with a bloom filter on v, then a is dropped so the current schema is [k, v] while the file keeps v at file position 2. The test rewrites the file index and asserts a side index file is written and holds the v index. Against the pre-#10122 code the rewrite failed here with IndexOutOfBoundsException.

API and Format

no

Documentation

no

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linked issue is a production correctness bug: after a column drop, rewrite_file_index can fail on old files or silently build an index under one column's name from another column's values. This change resolves the read projection in the current table schema while preserving the file-schema projection for the index writer, keeping their row positions aligned by field name. I checked dropped and re-added columns, current-schema consistency under concurrent DDL, and nested map-column de-duplication; no actionable regression was found.

The two new tests cover both the out-of-range failure and same-arity silent-corruption case, asserting actual bloom-filter values. They could not reach the rewritten path in my local Maven run because the test fixture failed while loading the existing CodeGenerator service (the same local failure occurs in unrelated core tests). The PR head's JDK 8/11, Flink 1/2, Spark, and E2E CI checks all pass.

@JingsongLi

Copy link
Copy Markdown
Contributor

Please rebase master.

apache#10122 fixed the rewrite_file_index read-projection drift after schema
evolution and shipped a test for the same-arity column-shift case. Add the
arity-shrink variant it did not cover: dropping a column before the indexed
one leaves a file-schema position past the current schema arity, which was
the IndexOutOfBoundsException path.
@LuciferYang
LuciferYang force-pushed the fix/file-index-rewrite-projection branch from 8a770d0 to 8c972df Compare September 24, 2026 14:50
@LuciferYang LuciferYang changed the title [core] Resolve rewrite_file_index read projection against current schema [core] Add regression test for rewrite_file_index over an arity-shrinking column drop Sep 24, 2026
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Repurposed to test-only. #10122 landed the same fix (reading the rewrite projection through the file schema), so I dropped the now-redundant production change and the test that duplicated its testRebuildsIndexOnCorrectColumnAfterColumnDropAndAdd. Force-pushed to rebase onto current master; what remains is one regression test, testProcessAfterDroppingColumnBeforeIndexedColumn, for the arity-shrinking column-drop case #10122 does not cover.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants