Skip to content

fix(core-initializers): do not clean up artefacts while the registry is being written - #6641

Open
delchev wants to merge 1 commit into
masterfrom
fix/synchronizer-cleanup-publish-race
Open

fix(core-initializers): do not clean up artefacts while the registry is being written#6641
delchev wants to merge 1 commit into
masterfrom
fix/synchronizer-cleanup-publish-race

Conversation

@delchev

@delchev delchev commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Fixes the intermittent IntentCrossModelFieldRetirementIT failure on master's postgres leg (e.g. run 31315813251).

What actually happens

A publish replaces a registry collection by deleting it and copying it back milliseconds later — both steps inside a single request. A synchronization pass that walks into that hole finds the sources gone and permanently deletes their artefact rows. From the failing run's log:

15:38:35.032  PublisherService   Unpublished collection: /registry/public/retire-owner/gen/invoices
15:38:35.676  JavaSynchronizer   Removing Java artefact [/retire-owner/gen/invoices/api/invoice/InvoiceController.java]
15:38:35.717  JavaSynchronizer   Removing Java artefact [/retire-owner/gen/invoices/data/invoice/InvoiceEntity.java]
   … four more …
15:38:35.736  PublisherService   Published collection: [/users/admin/workspace/retire-owner] → /registry/public/retire-owner

The files were back 21 ms after the last deletion, but their artefacts were gone. Reconcilers that rebuild from the whole artefact set then work from a half-empty registry — for client Java that is one javac batch over a source set whose cross-references cannot resolve:

Compiled batch: [8] units, [6] class file(s); [2] failed
   javac error … package gen.invoices.data.invoice does not exist
Compiled batch: [11] units, [0] class file(s); [11] failed   ← cascade of "cannot find symbol"

Zero bytecode, so the owner's InvoiceController was never registered and the test's 60 s poll saw nothing but 404s. Nothing re-triggered a rebuild, so the instance stayed dead.

The fix

Defer cleanup when a registry-mutating request was in flight while the pass ran. A new RegistryMutationTracker (core-base) counts those requests; a filter marks the publisher and workspace endpoints. Only mutating methods count — the IDE polls those endpoints with GET constantly — and the filter is registered explicitly through a FilterRegistrationBean so Boot does not map it to every request: a write through a generated application's REST controller is not a registry mutation.

The question is deliberately "was a client writing", not "did the registry change". My first attempt keyed on the file-system watcher flag, and SynchronizationInitializerDeletedTest caught it immediately: the watcher reports a genuine deletion exactly like a publish, so that version delayed every deletion by one pass. The tracker distinguishes the two with no timing heuristic at all.

Stop compiling a batch we already know is incomplete. JavaSynchronizer.rebuildAll used to silently skip a registered source whose file was momentarily missing and compile the rest; it now defers the rebuild to the next cycle, by when either the file is back or the orchestrator has dropped the artefact.

Rejected alternatives: a two-consecutive-passes rule delays every deletion (and breaks the delete-assertion tests), and a lock inside PublisherService cannot span the delete and the copy when a handler issues them separately.

Verification

SynchronizerCleanupRaceIT drives passes with a publish in flight. It does not assume a pass ran — a probe artefact published during the window must appear first, which is the proof a full pass completed (processSynchronizers() silently skips when another run holds the slot; an earlier version of this test passed against the unfixed code precisely because of that). It then asserts the missing source's artefact survived, and that a genuine deletion is still reconciled in a single forced call.

  • With the fix: SynchronizerCleanupRaceIT, IntentCrossModelFieldRetirementIT, RoleSynchronizerCleanupIT, JavaEngineIT (5/5), and core-initializers' unit tests (5/5, incl. SynchronizationInitializerDeletedTest) all green.
  • With the deferral disabled: SynchronizerCleanupRaceIT fails with [an artefact whose source vanished mid-publish must not be cleaned up] Expecting actual not to be empty.

mvn formatter:validate and the release-profile javadoc build pass on all touched modules.

🤖 Generated with Claude Code

…in flight

A publish replaces a registry collection by DELETING it and copying it back
milliseconds later, both inside a single request. A synchronization pass that
walks into that hole finds the sources gone and permanently deletes their
artefact rows - and the reconcilers that rebuild from the WHOLE artefact set
then work from a half-empty registry.

That is how master's postgres leg failed: a scheduled pass ran while
IntentCrossModelFieldRetirementIT published, deleted six client-Java artefacts
of a project whose files reappeared 21 ms later, and the batch compile that
followed produced ZERO class files ("package gen.invoices.data.invoice does not
exist", then a cascade of "cannot find symbol"). The owner's controller was
never registered, so the test's 60s poll saw nothing but 404s.

The pass now defers cleanup when a registry-mutating request was in flight
while it ran. It deliberately asks "was a client writing" and not "did the
registry change": the file-system watcher reports a genuine deletion exactly
like a publish, so keying on it delays EVERY deletion by a pass - which
SynchronizationInitializerDeletedTest caught. A new RegistryMutationTracker in
core-base counts those requests; a filter marks the publisher and workspace
endpoints (mutating methods only - the IDE polls them with GET constantly), and
it is registered explicitly so it never maps to a generated application's own
REST writes.

JavaSynchronizer stops compiling a batch it already knows is incomplete: a
registered source that is momentarily missing defers the rebuild to the next
cycle instead of submitting a source set whose cross-references cannot resolve.

SynchronizerCleanupRaceIT drives passes with a publish in flight and proves a
pass actually ran (a probe artefact must appear) before asserting the missing
source's artefact survived - and that a genuine deletion is still reconciled in
a single forced call. It fails without the deferral.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@delchev
delchev force-pushed the fix/synchronizer-cleanup-publish-race branch from f9ecdcf to 6e51fec Compare August 9, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant