[lake/iceberg] Add unit tests for FlussRecordAsIcebergRecord tiering wrapper#3574
Open
XuQianJin-Stars wants to merge 1 commit into
Open
[lake/iceberg] Add unit tests for FlussRecordAsIcebergRecord tiering wrapper#3574XuQianJin-Stars wants to merge 1 commit into
XuQianJin-Stars wants to merge 1 commit into
Conversation
…wrapper Introduce FlussRecordAsIcebergRecordTest that mirrors the coverage matrix of FlussRecordAsPaimonRowTest, focusing on behaviors specific to the tiering wrapper on top of FlussRowAsIcebergRecord: - All-types log-table row: dispatch of __bucket / __offset / __timestamp system columns via both positional get(pos) and name-based getField(name), alongside the existing 14 primitive/business columns delegated to the parent wrapper. - Primary-key table with all ChangeType values (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE / APPEND_ONLY): asserts that offset/timestamp are refreshed on every setFlussRecord call and system columns remain consistent. - Field count mismatch: setFlussRecord throws IllegalStateException when fluss row field count does not match struct.size - system. - Complex-type propagation via the parent converter: ARRAY<INT>, ARRAY<ARRAY<INT>>, ARRAY with nullable / empty / null variants, MAP<STRING,INT> populated / empty / null, and ROW<INT,STRING> including a null nested row. This mirrors the guardrails present in fluss-lake-paimon's FlussRecordAsPaimonRowTest so log-table tiering into Iceberg has equivalent unit-test coverage.
5c6d70f to
82e499c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce
FlussRecordAsIcebergRecordTestthat mirrors the coverage matrix ofFlussRecordAsPaimonRowTest, focusing on behaviors specific to the tiering wrapper on top ofFlussRowAsIcebergRecord:__bucket/__offset/__timestampsystem columns via both positionalget(pos)and name-basedgetField(name), alongside the existing 14 primitive/business columns delegated to the parent wrapper.ChangeTypevalues (INSERT/UPDATE_BEFORE/UPDATE_AFTER/DELETE/APPEND_ONLY): asserts thatoffset/timestampare refreshed on everysetFlussRecordcall and system columns remain consistent.setFlussRecordthrowsIllegalStateExceptionwhen Fluss row field count does not matchstruct.size - system.ARRAY<INT>,ARRAY<ARRAY<INT>>,ARRAYwith nullable / empty / null variants,MAP<STRING,INT>populated / empty / null, andROW<INT,STRING>including a null nested row.This mirrors the guardrails present in
fluss-lake-paimon'sFlussRecordAsPaimonRowTestso log-table tiering into Iceberg has equivalent unit-test coverage.Purpose
fluss-lake-iceberg's tiering wrapperFlussRecordAsIcebergRecordsits between the raw FlussLogRecordstream and Iceberg'sRecordinterface, injecting the three system columns (__bucket,__offset,__timestamp) on top of the field-by-field conversion done byFlussRowAsIcebergRecord. Prior to this PR, only the parent converter had unit-test coverage; the wrapper's system-column dispatch, field-count guardrail, per-recordoffset/timestamprefresh semantics and complex-type propagation were tested only indirectly through end-to-end IT cases. Meanwhile, the siblingfluss-lake-paimonmodule already carries a dedicatedFlussRecordAsPaimonRowTestthat exercises exactly this contract for its Paimon counterpart.This PR closes the gap by adding an Iceberg equivalent test class that pins down the same contract, so future refactors of the Iceberg tiering path (adding new complex types, changing system-column ordering, tightening the mismatch check, etc.) will surface regressions at unit-test speed rather than IT-test speed.
Linked issue: close #xxx
Brief change log
Test-only change. No production code is touched.
fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/tiering/FlussRecordAsIcebergRecordTest.java(+477 lines, 8@Testmethods):testLogTableRecordAllTypes— all-types (14 primitive/business columns + 3 system columns) log-table row; asserts both positionalget(pos)and name-basedgetField(name)for every column, including__bucket/__offset/__timestamp.testPrimaryKeyTableRecordAllChangeTypes— iterates overINSERT/UPDATE_BEFORE/UPDATE_AFTER/DELETE/APPEND_ONLY, callingsetFlussRecordwith distinct offsets/timestamps each time, and asserts the system columns are refreshed and business columns remain consistent per record.testFieldCountMismatchThrows— feeds a row whose field count does not matchstruct.size - systemand assertsIllegalStateExceptionis thrown bysetFlussRecord.testArrayTypeWithIntElements—ARRAY<INT>propagation through the parent converter.testNestedArrayType—ARRAY<ARRAY<INT>>propagation.testArrayWithNullableElementsAndEmptyAndNullArray—ARRAYwith nullable elements, empty array, and top-level null array.testMapTypeWithStringToIntAndEmptyAndNullMap—MAP<STRING,INT>populated / empty / null.testNestedRowType—ROW<INT,STRING>including a top-level null nested row.The 8 test methods together mirror the coverage matrix of
fluss-lake-paimon'sFlussRecordAsPaimonRowTest, adapted to the IcebergRecord/Types.NestedFieldAPI.Tests
New tests are the deliverable of this PR. Verified locally:
./mvnw -pl fluss-lake/fluss-lake-iceberg test -Dtest=FlussRecordAsIcebergRecordTest -Dspotless.check.skip— all 8 tests pass../mvnw -pl fluss-lake/fluss-lake-iceberg test -Dspotless.check.skip— full module test suite still green (no interference with existing tests)../mvnw -pl fluss-lake/fluss-lake-iceberg checkstyle:check— 0 violations.No production behavior change; existing tests are unchanged.
API and Format
None. This PR only adds tests under
src/test/java. There are no changes to public Java API, table properties, wire format, or Iceberg on-disk format.Documentation
None. Test-only change; no user-facing documentation impact.