Skip to content

feat/cq-calendar-month-intervals - #18547

Open
DaZuiZui wants to merge 10 commits into
apache:masterfrom
DaZuiZui:feat/cq-calendar-month-intervals
Open

feat/cq-calendar-month-intervals#18547
DaZuiZui wants to merge 10 commits into
apache:masterfrom
DaZuiZui:feat/cq-calendar-month-intervals

Conversation

@DaZuiZui

@DaZuiZui DaZuiZui commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Implementation

This change adds calendar-aware mo and y durations to Continuous Query EVERY and RANGE clauses.

  • Parse CQ durations into structured (monthPart, nonMonthDuration) values instead of flattening calendar units into fixed milliseconds.
  • Preserve structured durations when they are inherited from GROUP BY TIME.
  • Calculate CQ occurrences and RANGE boundaries from the original boundary in the persisted CQ time zone, applying calendar months first and fixed-duration components afterward.
  • Handle month-end clamping, leap years, DST transitions, omitted boundaries, and mixed calendar/fixed durations.
  • Add versioned structured duration fields to TCreateCQReq, while retaining legacy fixed-duration fields and compatibility handling for existing requests and snapshots.
  • Persist calendar-aware CQ metadata, boundary information, time zone, and occurrence progress across restart, leader recovery, and snapshot recovery.
  • Advance CQ progress using token/index validation so stale or duplicated callbacks cannot create competing schedules.
  • Calculate execution timeouts from the actual distance between adjacent calendar occurrences.
  • Add checked duration parsing/arithmetic, semantic validation, i18n messages, and focused tests for parsing, scheduling, persistence, recovery, and compatibility.

Fixes #18428

@DaZuiZui DaZuiZui changed the title feat: support calendar month and year CQ intervals feat/cq-calendar-month-intervals Aug 31, 2026
@DaZuiZui
DaZuiZui force-pushed the feat/cq-calendar-month-intervals branch from 2e70ead to b81d179 Compare September 4, 2026 03:21
@DaZuiZui
feat: support calendar month and year CQ intervals
d4ffaf8 
Treat CQ EVERY/RANGE mo/y as calendar months and years instead of
flattening them to 30d/365d. Reuse GROUP BY TIME calendar arithmetic,
fail closed on mixed-version clusters, and keep RANGE anchored to the
original boundary.

Fixes apache#18428
@DaZuiZui
DaZuiZui force-pushed the feat/cq-calendar-month-intervals branch from aeaad27 to d4ffaf8 Compare September 20, 2026 03:34
DaZuiZui and others added 9 commits September 20, 2026 14:57
@DaZuiZui
fix: recover legacy CQ snapshots and stale last-exec updates
b76fdec 
ReadWriteIOUtils.readInt throws a generic IOException at EOF, so
CQInfo now treats a short remaining stream as a pre-extension snapshot
instead of catching EOFException. Legacy CQs also reconcile durable
progress after CQ_UPDATE_LAST_EXEC_TIME_ERROR so they do not stall.
@DaZuiZui
fix: keep mixed CQ retry delay after recovery
c2a044e 
Fixed EVERY + calendar RANGE CQs persist everyInterval as 0.
Recovery must use calendarAware when computing retry delay so
restart/leader failover does not busy-spin on failure.
@DaZuiZui
test: cover calendar CQ natural-month windows and encoding rejects
7fe3c2b 
Add n=0 RANGE, DISCARD skip, mixed-version ingress, and an execution IT
that asserts EVERY 1mo RANGE 1mo uses the just-finished natural month.
@DaZuiZui
fix: harden CQ snapshot recovery stream and drop dead legacy branch
fac8039 
CQInfo.processLoadSnapshot used a raw FileInputStream, whose
available() does not contractually guarantee the remaining byte
count, for the extension-marker presence check in deserialize().
Read the snapshot into memory via ByteArrayInputStream instead,
matching the pattern already used in NodeInfo for the same kind of
optional-trailing-data probe.

Also remove the occurrenceIndex < 0 branch in
CQScheduleTask#occurrenceAt: every call site (both constructors and
persistProgress's legacy-path guard) only invokes it once
occurrenceIndex is already non-negative, so the branch was
unreachable.
@DaZuiZui
Remove out-of-scope mixed-version cluster capability negotiation from…
857388d 
… CQ calendar durations

Issue apache#18428 only asks for calendar-aware EVERY/RANGE duration support in
continuous queries. The previous commits on this branch additionally
introduced a cluster-wide node capability negotiation mechanism
(TNodeVersionInfo.supportedCQDurationEncodingVersions, snapshot/procedure/
plan serialization tails guarded by a magic marker, and "fail closed"
gating in NodeManager/CQManager/ClusterConfigTaskExecutor that rejects
calendar-duration CQs unless every node in the cluster advertises support).
That mechanism addresses a mixed-version-cluster rollout concern that the
issue never raised, and added a second, independent surface for bugs.

This trims the branch back to what the issue actually asks for: the
TCQDuration wire format, TCreateCQReq's structured duration fields, and
CQManager's structural/semantic validation stay; the capability-negotiation
scaffolding and its tests are removed.
@DaZuiZui
Revert "Remove out-of-scope mixed-version cluster capability negotiat…
76e59ce 
…ion from CQ calendar durations"

This reverts commit 857388d.
@DaZuiZui
test: add P0 coverage for calendar CQ ser/de and BOUNDARY 0 semantics
8f59c86 
Three critical paths were not covered by existing tests:

1. CreateCQProcedure serialization with calendar fields (everyDuration,
   boundaryExplicit) — the procedure recovery path via ProcedureFactory
   was never exercised with a calendar-aware CQ request.

2. UpdateCQLastExecTimePlan ser/de with occurrence index — the new
   5-parameter constructor (expectedIndex, targetIndex) had no round-trip
   test, and the legacy 3-parameter format (pre-calendar) had no forward
   compatibility test to confirm the optional tail is correctly skipped
   during Ratis log replay.

3. Explicit BOUNDARY 0 vs omitted BOUNDARY — the revised addendum promised
   'BOUNDARY 0 stays Unix epoch; omitted uses local 1970-01-01'. The
   implementation has 5 occurrences of isBoundaryExplicit() checks; flipping
   any ! operator would silently shift all BOUNDARY 0 CQs by the zone offset
   (e.g., 8 hours in Asia/Shanghai), but no test observed the difference.

The three new tests:
- calendarCqWithOmittedBoundarySurvivesSerializeDeserialize
- UpdateCQLastExecTimePlanWithOccurrenceIndexTest
- UpdateCQLastExecTimePlanLegacyFormatWithoutIndexTest
- testExplicitBoundaryZeroStaysUnixEpochWhileOmittedBoundaryUsesLocalEpoch

Mutation testing confirmed each catches the bug it was designed to prevent.

Related to apache#18428
@DaZuiZui
refactor: eliminate duplication and FQN pollution in CQ calendar code
126245e 
Before:
- TCQDuration→TimeDuration conversion (Math.toIntExact) repeated 7 times
- Boundary resolution (isSetBoundaryExplicit && !isBoundaryExplicit) repeated 5 times
- Precision scaling ("us"→1000, "ns"→1000000) repeated 3 times
- FQN pollution: 43 occurrences of org.apache.tsfile.utils.TimeDuration,
  java.time.ZoneId, org.apache.iotdb.common.rpc.thrift.T* in main code

After:
- New CQDurationUtils with toTimeDuration(), resolveBoundary(), currentTimeInPrecision()
- FQN reduced to 11 (1 intentional in CQDurationUtils, 10 unavoidable in lambdas/locals)
- Proper imports added: ZoneId, TConfigNodeLocation, TDataNodeConfiguration, Map

Other fixes:
- CQScheduleTask.run() wrapped with try-catch to prevent silent task death on assertion failure
- CreateContinuousQueryStatement.semanticCheck() reordered: positive-value check moved before
  lower-bound arithmetic (defensive programming)
- Typo: "deliberate not" → "deliberately not" in DataNodeDateTimeUtils

All 197 tests (175 ConfigNode + 22 DataNode) pass.

Related to apache#18428
@DaZuiZui
fix: address review findings on calendar CQ intervals
585afde 
- IoTDBCQExecIT: the calendar-month execution test returned silently when
  setup overran the scheduled first execution, producing a vacuous pass.
  Fail explicitly instead and widen the scheduling margin to 30s.
- ManagerMessages: rename EXCEPTION_CQ_OCCURRENCE_INDEX_DOES_NOT_MATCH_
  EXECUTION_TIME suffix from 7E4B91A2 to B2DE4B0F so it matches the md5 of
  the English value, per the i18n naming convention (en/zh in sync).
- CQCalendarUtils: drop unused nextAfter(); firstOccurrenceIndex covers it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support calendar-month/year (mo/y) intervals for Continuous Query EVERY and RANGE clauses

1 participant

@DaZuiZui