Conversation
…operator
sink.operator-uid.suffix reaches the writer, the global committer and the
dynamic bucket assigner; source.operator-uid.suffix reaches the source.
Every other operator Paimon adds takes its id from the shape of the stream
graph, so a change elsewhere in the job orphans its checkpoint entry.
Recovery from the HA checkpoint store then fails with "There is no
operator for the state <id>".
Add sink.operator-uid.cover-all-operators and
source.operator-uid.cover-all-operators, off by default. With the option
and the matching suffix set, every operator a streaming read or write
adds gets ${prefix}_${table}_${suffix}. Existing uids are unchanged.
Batch-only topologies (clustering, sort compaction, postpone
merge-on-read) never restore from a checkpoint and are left alone.
Tests: a sink and a source graph matrix built under
pipeline.auto-generate-uids=false, so Flink itself rejects any uid-less
operator per topology shape; literal pins for every pre-existing and
new uid; an ITCase restoring from a retained checkpoint store after an
upstream topology change; an ITCase migrating from the old uid layout
through execution.state-recovery.path, including PARTITION_DYNAMIC.
Docs: a migration guide under docs/docs/flink/savepoint.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #10107.
sink.operator-uid.suffixandsource.operator-uid.suffixreach only Writer, Global Committer, dynamic-bucket-assigner and Source. Every other operator a streaming Paimon read or write adds takes its id from the stream graph shape, so any topology change orphans its checkpoint entry, and a restore from the HA checkpoint store fails withThere is no operator for the state <id>. Discussed on dev@: [DISCUSS] The operator-uid.suffix options do not cover every operator.This adds
sink.operator-uid.cover-all-operatorsandsource.operator-uid.cover-all-operators, boolean, default false. With the option and the matching suffix set, every streaming operator gets<prefix>_<table>_<suffix>. Existing uids and the default graph are unchanged. Batch-only operators are out of scope: they never restore from a checkpoint.Things a reviewer should know:
Collect Statistics(aCoordinatedOperatorFactory, present underpartition.sink-strategy = PARTITION_DYNAMIC) and theMonitorsource on the dedicated-split-generation and exactly-once-consumer routes. On the explicit-path route those two needexecution.state-recovery.ignore-unclaimed-stateonce; a recovery from the HA checkpoint store cannot skip entries at all, so the migration goes through a savepoint or an explicit checkpoint path. The guide indocs/docs/flink/savepoint.md("Operator UIDs") spells this out and shows that source, writer and committer state stays mapped.endsink is covered. The HA route rejects any unclaimed entry regardless of content,endacknowledges checkpoints and so has one, and SQL users have no way to uid it themselves. It only changes with the option on, and a DataStream user's own.uid()on the returned sink still wins. Open for discussion, as is the option name.db1.ordersanddb2.orderswritten with one suffix collide onWriter_orders_s. This is pre-existing for the four old uids and now applies to about twenty; the guide tells users to use a distinct suffix per table.*_NAMEconstants, followingWRITER_NAME,GLOBAL_COMMITTER_NAMEandSOURCE_NAME. They are pinned literally in tests.Tests
OperatorUidSuffixTest(17) andSourceOperatorUidSuffixTest(12): 12 sink and 7 source topology shapes each built underpipeline.auto-generate-uids=false, so Flink itself rejects any uid-less operator; exact node-name guards per shape; literal pins for the four pre-existing uids and for every new one; option off and option-without-suffix leave today's uids only; operator ids survive an upstream topology change.OperatorUidSuffixRestoreITCase(2): restore from a retained checkpoint store (HA route,EmbeddedHaServicesWithLeadershipControl+PerJobCheckpointRecoveryFactory) after adding an upstream operator succeeds with the option on and fails withThere is no operator for the statewith it off.OperatorUidMigrationITCase(2): checkpoint taken without the option, restored with it throughexecution.state-recovery.path. Unaware-bucket succeeds. PARTITION_DYNAMIC fails withCannot map checkpoint/savepoint state for operator, then succeeds withexecution.state-recovery.ignore-unclaimed-state; the restored job commits under the samecommitUser(committer state mapped) and the dropped entries, read from the checkpoint metadata, are exactly the previously uid-less operators.paimon-flink-commonunit phase on Flink 1.20.1 (1094 tests), the four classes above on Flink 2.2.0 (-Pflink2),paimon-flink-cdccompile,ConfigOptionsDocsCompletenessITCase, and-DskipTests verify(checkstyle, spotless, rat, enforcer). Not built: thepaimon-flink-1.16..2.1version modules andpaimon-flink-action.🤖 Generated with Claude Code