Skip to content

[server] Support update_if_changed merge engine for primary-key tables - #4439

Open
litiliu wants to merge 1 commit into
apache:mainfrom
litiliu:feature/update-if-changed-merge-engine
Open

litiliu wants to merge 1 commit into
apache:mainfrom
litiliu:feature/update-if-changed-merge-engine

Conversation

@litiliu

@litiliu litiliu commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduce an UPDATE_IF_CHANGED merge engine for primary-key tables ('table.merge-engine' = 'update_if_changed').

It keeps last-row upsert semantics but suppresses value-identical writes:

Stored row Incoming operation Result
Absent Insert/upsert Store the row, emit an insert changelog record
Present Every logical field is equal Keep the stored row, emit no changelog record (no-op)
Present At least one field differs Store the row, emit a normal update changelog record
Present Delete Delete the row, emit a delete changelog record
Absent Delete No-op

Equality is based on logical field values, not raw bytes: null values use logical equality, binary values are compared by content, and rows from different schema versions are aligned to the latest schema by stable column IDs. Fields that exist in the latest schema but are absent from either compared row are treated as null. Partial updates and partial deletes are first applied to the stored row to produce a complete candidate row, which is then compared with the stored row.

Because it must read the stored value before deciding, it does not use the WAL full-row fast path that skips old-value lookup (the merger is not a DefaultRowMerger, so canSkipOldValueLookup stays false).

Fixes #4343

Changes

  • MergeEngineType.UPDATE_IF_CHANGED + fromString and config description
  • New UpdateIfChangedRowMerger, supporting full-row, partial-update, and partial-delete mutations with logical row equality, wired into RowMerger.create()
  • Schema-version alignment by stable column IDs, including schemas with dropped columns
  • FlinkTableSink: allow partial updates and UPDATE/DELETE for this engine
  • Docs: new merge-engine page + listings/options updates

Test Plan

  • UpdateIfChangedRowMergerTest and RowMergerCreateTest (unit: logical equality, schema alignment, full-row and partial mutations, and merger creation)
  • FlussTableITCase#testUpdateIfChangedMergeEngine (end-to-end: value-identical full-row upserts emit no changelog record while changes and deletes do)
  • FlussTableITCase#testUpdateIfChangedMergeEngineWithPartialUpdate (end-to-end: value-identical partial updates/deletes are no-ops while changed mutations emit changelog records)
  • Both client end-to-end tests run with FULL and WAL changelog images
  • FlinkTableSinkITCase#testUpdateIfChangedMergeEngineSupportsMutations (Flink sink integration: partial INSERT, UPDATE, and DELETE)

🤖 AI-assisted changes - reviewed by human developer

@litiliu
litiliu marked this pull request as draft September 20, 2026 06:38
@litiliu
litiliu force-pushed the feature/update-if-changed-merge-engine branch 7 times, most recently from 9ebf163 to 5da67ad Compare September 20, 2026 10:03
@litiliu
litiliu force-pushed the feature/update-if-changed-merge-engine branch from 5da67ad to 5a7eec9 Compare September 21, 2026 03:22
@litiliu
litiliu marked this pull request as ready for review September 21, 2026 06:05
@litiliu

litiliu commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

@luoyuxia PTAL, when you have time thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support an update-if-changed merge engine for primary-key tables

1 participant