Skip to content

[refactor](ivm) Rename IvmInfo.refreshVersion to sequencePrefix - #68336

Open
yujun777 wants to merge 1 commit into
apache:masterfrom
yujun777:ivm-sequence-prefix-rename
Open

yujun777 wants to merge 1 commit into
apache:masterfrom
yujun777:ivm-sequence-prefix-rename

Conversation

@yujun777

Copy link
Copy Markdown
Contributor

IvmInfo.refreshVersion is the high part of the sequence values an IVM MV's rows are stamped with -- the low part next to it is a delta index. The name reads like an epoch or like the MV's own version, which is what the per-partition refreshEpoch being added alongside it is not, and the two sitting in the same code base is a trap for the next reader.

This renames it to sequencePrefix: the prefix of the (sequence prefix, delta index, op) triple that IvmSequenceCalculator encodes into the sequence column. Nothing about the value changes -- it still counts committing IVM transactions and still prefixes the sequence column.

  • field and accessors: sequencePrefix, getSequencePrefix(), advanceSequencePrefix()
  • MTMV.getNextRefreshVersion() -> MTMV.getNextSequencePrefix()
  • IvmSequenceCalculator identifiers, including LARGEINT_SEQUENCE_PREFIX_SHIFT and the range-check messages
  • the persisted name changes outright from "rv" to "sp": IVM is not released, so there is no image or journal in the wild that writes the old name

Trace: #65418

The field is the high part of the sequence values an IVM MV's rows are stamped with -- the low part next
to it is a delta index -- but "refreshVersion" reads like an epoch or like the MV's own version, which is
exactly what the per-partition refreshEpoch being added alongside it is not. Nothing about the value
changes: it still counts committing IVM transactions and still prefixes the sequence column.

Key changes:
- Rename the field and its accessors to sequencePrefix / getSequencePrefix / advanceSequencePrefix, and MTMV.getNextRefreshVersion to getNextSequencePrefix
- Rename the identifiers in IvmSequenceCalculator to match, including LARGEINT_SEQUENCE_PREFIX_SHIFT and the range-check messages
- Persist it as "sp" instead of "rv": IVM is not released, so there is no image or journal that writes the old name

Unit Test:
- IvmInfoTest.testSequencePrefixIsPersistedAsSp pins the persisted name and that "rv" is gone
- IvmInfoTest / IvmSequenceCalculatorTest / IvmAggDeltaHandlerTest / IvmDeltaRewriteStateTest / DatabaseTransactionMgrTest cover the renamed API
@yujun777
yujun777 requested a review from morrySnow as a code owner September 21, 2026 10:19
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review result: one terminology-completeness issue remains; no runtime correctness defect was found.

  • Goal and scope: the production change is behavior-preserving after normalizing the renamed identifiers. The accessor, rewrite state, calculator, commit/replay advancement, tests, and new sp serialization name are internally consistent. The inline comment identifies the remaining old terminology that conflicts with the PR's stated clarity goal.
  • Concurrency and lifecycle: no lock, shared-state, or ordering behavior changes. Local commit and COMMITTED replay still advance exactly once under existing table/transaction locking; VISIBLE replay does not advance again. Dry run and EXPLAIN remain read-only.
  • Persistence and compatibility: current images and AlterMTMV journals consistently persist sp; checkpoint/restart/failover paths were traced. Old rv or mixed-version IVM metadata would reset the prefix, but the repository's explicit pre-October-2026 IVM policy permits this pre-release incompatibility.
  • Parallel paths and encoding: local and cloud prefix ownership, linear/join/aggregate consumers, BIGINT/LARGEINT bit layouts and bounds, error paths, performance, and observability are unchanged. No config, FE-BE protocol, or additional storage-format change is introduced.
  • Tests and validation: the changed unit tests retain exact encoding/boundary assertions and cover Gson/copy, normal commit, and replay. This review was static-only as required; no builds or tests were run here. At submission time, CheckStyle and compile checks were passing, while FE UT and regression jobs were still pending.
  • User focus: no additional focus was provided; the complete PR was reviewed.

### Semantics

- **Read-only.** No insert transaction is built. Stream offsets, refresh version, and MV metadata
- **Read-only.** No insert transaction is built. Stream offsets, sequence prefix, and MV metadata

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you finish the terminology sweep? regression-test/plugins/plugin_planner.groovy:138 and regression-test/suites/mtmv_p0/ivm/test_ivm_refresh_dry_run.groovy:19 still say the sequence encodes a "refresh version", and the renamed transaction test still uses *_refresh_version_* fixture names. Since this refactor is specifically meant to remove that ambiguity, leaving these references keeps teaching the old concept.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 73d08f9ee22ba4163e3165249d84c92022238a18, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17598	3748	3729	3729
q2	2184	351	303	303
q3	10072	1382	795	795
q4	4686	468	345	345
q5	7461	822	558	558
q6	183	169	133	133
q7	748	778	600	600
q8	9301	1438	1396	1396
q9	5428	4116	4108	4108
q10	6839	1302	1032	1032
q11	429	264	240	240
q12	633	411	292	292
q13	18082	2586	1972	1972
q14	262	260	233	233
q15	q16	723	710	651	651
q17	1678	1134	978	978
q18	6512	5601	5538	5538
q19	1372	1247	1067	1067
q20	474	390	251	251
q21	5794	2923	2677	2677
q22	420	347	296	296
Total cold run time: 100879 ms
Total hot run time: 27194 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4117	4008	4022	4008
q2	707	554	522	522
q3	4451	4874	4290	4290
q4	2199	2321	1436	1436
q5	4166	4053	4093	4053
q6	217	168	123	123
q7	1729	1589	1407	1407
q8	2174	1885	2316	1885
q9	7570	7227	7306	7227
q10	3694	3609	3174	3174
q11	555	402	374	374
q12	713	737	510	510
q13	2478	2742	2158	2158
q14	308	290	276	276
q15	q16	702	729	638	638
q17	7886	7240	7181	7181
q18	11859	11080	11843	11080
q19	1212	1090	1075	1075
q20	2224	2194	1938	1938
q21	5276	4504	4544	4504
q22	514	434	419	419
Total cold run time: 64751 ms
Total hot run time: 58278 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152307 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 73d08f9ee22ba4163e3165249d84c92022238a18, data reload: false

query5	4346	619	474	474
query6	453	219	208	208
query7	4827	542	296	296
query8	317	182	167	167
query9	8825	3938	4010	3938
query10	450	312	253	253
query11	5904	3547	3226	3226
query12	146	94	88	88
query13	1289	597	410	410
query14	6544	4509	4221	4221
query14_1	3990	3960	3933	3933
query15	197	188	169	169
query16	1000	452	348	348
query17	881	635	531	531
query18	2453	443	327	327
query19	202	174	134	134
query20	86	83	78	78
query21	222	131	114	114
query22	13110	13004	12864	12864
query23	13942	12842	12365	12365
query23_1	12541	12414	12430	12414
query24	7327	1103	667	667
query24_1	635	703	694	694
query25	529	409	351	351
query26	1246	316	161	161
query27	2715	579	337	337
query28	4520	1963	1959	1959
query29	1580	707	507	507
query30	296	215	181	181
query31	887	759	634	634
query32	142	89	96	89
query33	499	300	257	257
query34	1155	1085	641	641
query35	737	754	642	642
query36	785	784	705	705
query37	144	102	91	91
query38	1834	1758	1690	1690
query39	683	669	669	669
query39_1	647	644	660	644
query40	221	123	106	106
query41	72	70	69	69
query42	96	93	98	93
query43	331	342	303	303
query44	1372	717	714	714
query45	184	177	163	163
query46	1084	1209	692	692
query47	1486	1475	1377	1377
query48	401	407	287	287
query49	589	391	295	295
query50	946	360	243	243
query51	10388	10676	10590	10590
query52	89	92	72	72
query53	237	246	179	179
query54	261	209	186	186
query55	77	72	67	67
query56	221	197	220	197
query57	1510	1432	1423	1423
query58	282	256	257	256
query59	1974	2053	1857	1857
query60	285	241	220	220
query61	145	139	141	139
query62	398	324	260	260
query63	209	172	172	172
query64	2805	1001	870	870
query65	3471	3395	3422	3395
query66	1808	407	294	294
query67	19969	19805	19924	19805
query68	3267	1583	1011	1011
query69	417	305	247	247
query70	895	778	784	778
query71	305	235	213	213
query72	2728	2601	2361	2361
query73	850	815	446	446
query74	4629	4451	4300	4300
query75	2310	2288	1954	1954
query76	2349	1085	720	720
query77	361	399	355	355
query78	9115	9002	8410	8410
query79	1339	1148	708	708
query80	576	456	366	366
query81	540	315	284	284
query82	639	168	123	123
query83	307	221	192	192
query84	327	141	114	114
query85	846	453	377	377
query86	335	243	230	230
query87	1991	1965	1816	1816
query88	3594	2706	2707	2706
query89	356	277	244	244
query90	1929	180	180	180
query91	167	157	125	125
query92	101	82	91	82
query93	1426	1396	828	828
query94	514	351	287	287
query95	672	365	412	365
query96	1039	792	360	360
query97	2410	2412	2308	2308
query98	161	149	142	142
query99	722	728	600	600
Total cold run time: 235958 ms
Total hot run time: 152307 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 73d08f9ee22ba4163e3165249d84c92022238a18, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.27	0.14	0.13
query4	1.61	0.14	0.14
query5	0.24	0.22	0.21
query6	1.15	0.95	0.89
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.38	0.33	0.34
query10	0.54	0.53	0.55
query11	0.20	0.16	0.15
query12	0.18	0.15	0.14
query13	0.46	0.47	0.48
query14	0.95	0.95	0.95
query15	0.59	0.58	0.57
query16	0.31	0.31	0.32
query17	1.04	1.09	1.04
query18	0.21	0.20	0.20
query19	1.97	1.89	1.97
query20	0.02	0.01	0.02
query21	15.48	0.20	0.14
query22	4.98	0.05	0.05
query23	16.15	0.31	0.12
query24	2.90	0.42	0.32
query25	0.10	0.06	0.05
query26	0.73	0.21	0.15
query27	0.05	0.04	0.03
query28	3.58	0.78	0.35
query29	12.47	4.06	3.20
query30	0.28	0.15	0.15
query31	2.77	0.54	0.31
query32	3.23	0.59	0.49
query33	3.25	3.24	3.15
query34	15.49	3.93	3.30
query35	3.25	3.22	3.22
query36	0.54	0.44	0.42
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.03	0.03	0.04
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.04
query43	0.05	0.04	0.03
Total cold run time: 96.11 s
Total hot run time: 23.85 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants