fix(sandbox): serialize same-key concurrent calls to prevent persisted state races - #2846
Open
larry-zy wants to merge 4 commits into
Open
fix(sandbox): serialize same-key concurrent calls to prevent persisted state races#2846larry-zy wants to merge 4 commits into
larry-zy wants to merge 4 commits into
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
guslegend0510
suggested changes
Aug 26, 2026
larry-zy
force-pushed
the
fix/same-key-acquire-serialization
branch
from
August 26, 2026 09:08
d879360 to
600631e
Compare
…state races Two concurrent HarnessAgent calls resolving to the same SandboxIsolationKey shared one persisted state slot. The acquire/resume/persist/release window runs outside the delegate's serializeOnKey gate and the sandbox guard defaulted to noop(), so both calls started containers from the same state and overwrote each other on completion (last write wins). - Add JVM-local InProcessSandboxExecutionGuard (fair per-key Semaphore; not thread-ownership bound so acquire/release may cross threads) and make it the default instead of noop(). - Run sandbox acquire/release on boundedElastic since the guard now blocks; a busy slot must never stall the subscriber's event-loop thread. - Release the guard lease in a finally and restore the interrupt flag cleared by InterruptedException; add inProcess(Duration) as a wedged-holder backstop.
larry-zy
force-pushed
the
fix/same-key-acquire-serialization
branch
from
August 26, 2026 09:21
600631e to
bba295d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2845
Background
On a single
HarnessAgent, two concurrent calls that resolve to the sameSandboxIsolationKeyread and write the same persisted sandbox state record.The sandbox
acquire → resume → persist → releasewindow runs outside the delegate'sserializeOnKeygate, and the sandbox-layer guard defaulted tonoop()— so there was no serialisation. Both calls started independent containers from the same state, mutated them in parallel, and overwrote each other on completion: workspace changes are silently lost (last write wins), with a transient double-container in between.This is not covered by #2490. That fix isolates the in-memory live binding for different-key concurrent calls; it does not stop same-key calls from racing on the persisted state.
Changes
InProcessSandboxExecutionGuard(JVM-local default guard): serialises perSandboxIsolationKey, one fairSemaphore(1)per key.Semaphorerather thanReentrantLockbecause acquire happens in the reactive resource supplier and release in the corresponding cleanup, which may run on different threads —Semaphoreis not thread-ownership bound.noop()toinProcess()(HarnessAgent): single-instance deployments serialise out of the box; multi-instance deployments can still supply a distributed guard viaSandboxFilesystemSpec#executionGuardor aDistributedStore.boundedElasticthread: the guard now genuinely blocks (potentially for the full duration of a same-slot peer call), so it must never tie up a WebFlux event-loop thread and starve unrelated sessions.finally(SandboxLifecycleMiddleware):lease.close()is guaranteed regardless of what persist/release throw, preventing a permit leak that would block every future same-slot call forever; also restores the interrupt flag cleared byInterruptedException.inProcess(Duration)as a backstop against a wedged holder, throwingSandboxExecutionTimeoutExceptionon expiry. This is a wedged-holder backstop, not a contention timeout — set it well above the maximum realistic call duration.Tests
InProcessSandboxExecutionGuardTest— queued serialisation, timeout, no slot leak, cross-thread release.SandboxSameSessionSerializationTest— same-key concurrent calls no longer overwrite each other end-to-end.HarnessAgentSandboxOffThreadTest— acquire/release never run on the subscriber thread.Deployment note
This guard coordinates within a single JVM only. When the same slot can be contended across multiple instances, supply a distributed implementation (e.g. Redis
SET NX).