[flink] Fix batch database compaction for Data Evolution tables - #10110
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The linked #10102 failure is a real end-to-end issue: batch compact_database routed Data Evolution partial-column files into ordinary append compaction, which cannot reconstruct a required column. This change routes those tables through the existing DataEvolutionTableCompact path while retaining ordinary tables in the combined path. I checked both routing modes, table selection, snapshot handling, and the new regression assertions; I found no actionable correctness issue.
Local verification: all 11 new integration-test invocations passed (six testDataEvolutionTableCompact combinations, two Data Evolution-only cases, two filter cases, and the streaming rejection case). The initial sandbox run could not bind Flink's local BLOB Server. On rerun, the local Flink test classpath selected Avro 1.11.3 and failed with DataFileWriter.setEncoder missing before exercising the change; temporarily selecting the project's Avro 1.11.4 for the isolated test run resolved this. The temporary POM adjustment was reverted and the worktree is clean. CI is green.
Requirement fit: SUPPORTED. Implementation: CLEAN for the batch behavior covered here.
Purpose
Fixes #10102.
Fix batch
compact_databaserouting for Data Evolution tables.The ordinary append compaction path reads partial-column files separately,
which can fail when a file omits a NOT NULL column.
Reuse
DataEvolutionTableCompactin divided mode. In batch combined mode,route Data Evolution tables to dedicated compactors while keeping other
tables on the existing combined path.
Tests
Add regression coverage in
CompactDatabaseActionITCasefor: