Skip to content

[flink] Fix batch database compaction for Data Evolution tables - #10110

Merged
JingsongLi merged 1 commit into
apache:masterfrom
tzphh:codex/fix-data-evolution-db-compaction
Sep 24, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
tzphh:codex/fix-data-evolution-db-compaction

Conversation

@tzphh

@tzphh tzphh commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Purpose

Fixes #10102.

Fix batch compact_database routing for Data Evolution tables.
The ordinary append compaction path reads partial-column files separately,
which can fail when a file omits a NOT NULL column.

Reuse DataEvolutionTableCompact in divided mode. In batch combined mode,
route Data Evolution tables to dedicated compactors while keeping other
tables on the existing combined path.

Tests

Add regression coverage in CompactDatabaseActionITCase for:

  • Partial-column updates that omit a NOT NULL column.
  • Batch divided and combined modes.
  • Mixed ordinary append and Data Evolution tables.
  • Data Evolution-only databases, empty tables, and table filtering.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linked #10102 failure is a real end-to-end issue: batch compact_database routed Data Evolution partial-column files into ordinary append compaction, which cannot reconstruct a required column. This change routes those tables through the existing DataEvolutionTableCompact path while retaining ordinary tables in the combined path. I checked both routing modes, table selection, snapshot handling, and the new regression assertions; I found no actionable correctness issue.

Local verification: all 11 new integration-test invocations passed (six testDataEvolutionTableCompact combinations, two Data Evolution-only cases, two filter cases, and the streaming rejection case). The initial sandbox run could not bind Flink's local BLOB Server. On rerun, the local Flink test classpath selected Avro 1.11.3 and failed with DataFileWriter.setEncoder missing before exercising the change; temporarily selecting the project's Avro 1.11.4 for the isolated test run resolved this. The temporary POM adjustment was reverted and the worktree is clean. CI is green.

Requirement fit: SUPPORTED. Implementation: CLEAN for the batch behavior covered here.

@JingsongLi
JingsongLi merged commit f7e9614 into apache:master Sep 24, 2026
22 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Flink compact_database uses ordinary append compaction for Data Evolution tables

2 participants