Preserve workflow-to-CHASM backfill migration safety

[chaptersix]

Problem

Workflow-to-CHASM migration can replay an ongoing backfill boundary. Independent review also exposed two related transfer hazards: a valid Unix-epoch cursor is indistinguishable from an unset watermark, and migrated backfillers can process imported buffered starts before stale running-workflow entries are reconciled.

Evidence: #42.

Violated invariants

• Preserve the remaining backfill occurrence set (exclusive cursor, inclusive end].
• Keep every migrated backfiller's first task valid and retryable.
• Do not evaluate overlap policy against stale imported running-workflow state.

Before / after

Before:

• V1 cursor 11:59, end 12:00 becomes a fresh CHASM range and emits [11:59, 12:00].
• Setting Attempt=1 to represent progress would strand the initial TaskStamp=1 task.
• A migrated backfiller starts during creation and can drop pending work under SKIP before callback reconciliation.

After:

• StartTime is cloned into LastProcessedTime, Attempt remains zero, and an explicit progress bit represents the Unix epoch without changing legacy nonzero-watermark behavior.
• Migrated backfillers with stale running entries are created inert and scheduled once by the existing callback task after reconciliation.
• Fresh, progressed, exact-boundary, capacity-stalled, empty, epoch, task-validity, and callback-barrier cases pass.

Implementation

• convertBackfillsLegacyToCHASM preserves the exclusive cursor and explicit progress state.
• BackfillerTaskHandler retains backward-compatible nonzero-watermark inference and records progress after committed partial work.
• CreateSchedulerFromMigration and SchedulerCallbacksTaskHandler defer and idempotently start migrated backfillers around callback reconciliation.
• Regression matrix and engine controls
• Canonical bug note

Compatibility and operations

The new proto bit is additive. New binaries infer progress from older nonzero watermarks; fresh and capacity-stalled old state stays fresh. The migration attempt remains zero so the initial task satisfies TaskStamp > Attempt. Deploy readers/writers before relying on the epoch distinction during rolling upgrades.

The change adds no ordinary scheduler hot-path read or write. At 10x migration load it copies one timestamp/bit per ongoing backfill and reuses the already-required callback transition. Crash retries reconstruct the same cursor; callback retries do not duplicate the initial task because a nonzero task stamp is retained.

Validation

go test -tags test_dep ./chasm/lib/scheduler ./chasm/lib/scheduler/migration ./service/worker/scheduler -count=1
go test -tags test_dep ./chasm/lib/scheduler/migration -run '^$' \
  -fuzz '^FuzzLegacyBackfillCursorMigration$' -fuzztime=3s
make proto
make lint-code-fast GOLANGCI_LINT_BASE_REV=chaptersix/main GOLANGCI_LINT_FIX=false

Fuzz seed corpus: (0,0), (-60,999000000), (86400,1); final bounded run executed 144,707 inputs and passed. Proto regeneration was clean and lint reported zero issues.

Stack

• Parent: Prove workflow-to-CHASM backfill cursor replay #42
• Base branch: migration-backfill/evidence

Rendered failure/sequence artifacts are attached below.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-05T01:06:05.182456Z	ad21de7	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad21de785c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not persist taskless backfillers during mixed-version rollout

During a rolling upgrade or shard failover, a new server can take this branch and persist migrated backfillers without tasks, after which the SchedulerCallbacksTask may execute on an older server whose handler only rearms the invoker and generator. Once that handler marks every imported start as having a callback, the callback task becomes invalid, leaving these backfillers permanently present but never executable. The taskless representation must not be written until every possible task processor knows how to arm it, or the deferred state must be recoverable independently of the new callback-handler behavior.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Stage epoch-progress semantics before writing the new bit

When an epoch cursor is migrated during a rolling upgrade or later handled after rollback, an older BackfillerTaskHandler ignores this unknown proto bit and still interprets the (0,0) LastProcessedTime as an unset watermark. It consequently subtracts one millisecond from the request start and emits the occurrence at the Unix epoch again. Writing state whose correctness depends on this field therefore needs a staged/version-gated rollout so old task readers cannot process it.

Useful? React with 👍 / 👎.