[tiering] Optimize tiering split queue and commit collection - #4457
Conversation
|
Benchmarked the before/after bookkeeping logic with JMH 1.37 and 1,000 splits per table, restricted to one logical CPU ( Environment: Linux/arm64 on Apple M3 Pro, JDK 17.0.18, G1, 512 MB heap, one benchmark thread. Two forks, each with 2 × 1 s warmup and 3 × 1 s measurement.
Values are mean times from standalone copies of the changed logic. The benchmark excludes Flink scheduling, logging, RPC and lake I/O; these results do not represent end-to-end tiering speedups. |
|
@luoyuxia cc |
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
No unresolved issues were identified that would block approval.
Review effort: Lite
Findings: None
What changed in this PR
Optimizes tiering bookkeeping from quadratic to linear work while preserving split ordering and commit validation.
Changes:
- Replaces pending split list removal with FIFO deque polling.
- Adds incremental per-table result accumulation with deduplication and count validation.
- Expands tests for ordering, duplicate results, inconsistent counts, and failed-round cleanup.
| File | Description |
|---|---|
| fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumeratorTest.java | Updated as part of this pull request. |
| fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumeratorSchedulingTest.java | Updated as part of this pull request. |
| fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/committer/TieringCommitOperatorTest.java | Updated as part of this pull request. |
| fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/enumerator/TieringSourceEnumerator.java | Updated as part of this pull request. |
| fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/committer/TieringCommitOperator.java | Updated as part of this pull request. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Purpose
Linked issue: close #4455
Brief change log
Tests
API and Format
Documentation