Skip to content

fix(sandbox): serialize same-key concurrent calls to prevent persisted state races - #2846

Open
larry-zy wants to merge 4 commits into
agentscope-ai:mainfrom
larry-zy:fix/same-key-acquire-serialization
Open

fix(sandbox): serialize same-key concurrent calls to prevent persisted state races#2846
larry-zy wants to merge 4 commits into
agentscope-ai:mainfrom
larry-zy:fix/same-key-acquire-serialization

Conversation

@larry-zy

@larry-zy larry-zy commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Closes #2845

Background

On a single HarnessAgent, two concurrent calls that resolve to the same SandboxIsolationKey read and write the same persisted sandbox state record.

The sandbox acquire → resume → persist → release window runs outside the delegate's serializeOnKey gate, and the sandbox-layer guard defaulted to noop() — so there was no serialisation. Both calls started independent containers from the same state, mutated them in parallel, and overwrote each other on completion: workspace changes are silently lost (last write wins), with a transient double-container in between.

This is not covered by #2490. That fix isolates the in-memory live binding for different-key concurrent calls; it does not stop same-key calls from racing on the persisted state.

Changes

  • Add InProcessSandboxExecutionGuard (JVM-local default guard): serialises per SandboxIsolationKey, one fair Semaphore(1) per key. Semaphore rather than ReentrantLock because acquire happens in the reactive resource supplier and release in the corresponding cleanup, which may run on different threads — Semaphore is not thread-ownership bound.
  • Flip the default guard from noop() to inProcess() (HarnessAgent): single-instance deployments serialise out of the box; multi-instance deployments can still supply a distributed guard via SandboxFilesystemSpec#executionGuard or a DistributedStore.
  • Move sandbox acquire/release onto a boundedElastic thread: the guard now genuinely blocks (potentially for the full duration of a same-slot peer call), so it must never tie up a WebFlux event-loop thread and starve unrelated sessions.
  • Release the lease in a finally (SandboxLifecycleMiddleware): lease.close() is guaranteed regardless of what persist/release throw, preventing a permit leak that would block every future same-slot call forever; also restores the interrupt flag cleared by InterruptedException.
  • Add inProcess(Duration) as a backstop against a wedged holder, throwing SandboxExecutionTimeoutException on expiry. This is a wedged-holder backstop, not a contention timeout — set it well above the maximum realistic call duration.

Tests

  • InProcessSandboxExecutionGuardTest — queued serialisation, timeout, no slot leak, cross-thread release.
  • SandboxSameSessionSerializationTest — same-key concurrent calls no longer overwrite each other end-to-end.
  • HarnessAgentSandboxOffThreadTest — acquire/release never run on the subscriber thread.

Deployment note

This guard coordinates within a single JVM only. When the same slot can be contended across multiple instances, supply a distributed implementation (e.g. Redis SET NX).

@codecov

codecov Bot commented Aug 26, 2026

Copy link
Copy Markdown

@larry-zy
larry-zy force-pushed the fix/same-key-acquire-serialization branch from d879360 to 600631e Compare August 26, 2026 09:08
…state races

Two concurrent HarnessAgent calls resolving to the same SandboxIsolationKey
shared one persisted state slot. The acquire/resume/persist/release window runs
outside the delegate's serializeOnKey gate and the sandbox guard defaulted to
noop(), so both calls started containers from the same state and overwrote each
other on completion (last write wins).

- Add JVM-local InProcessSandboxExecutionGuard (fair per-key Semaphore; not
  thread-ownership bound so acquire/release may cross threads) and make it the
  default instead of noop().
- Run sandbox acquire/release on boundedElastic since the guard now blocks; a
  busy slot must never stall the subscriber's event-loop thread.
- Release the guard lease in a finally and restore the interrupt flag cleared by
  InterruptedException; add inProcess(Duration) as a wedged-holder backstop.
@larry-zy
larry-zy force-pushed the fix/same-key-acquire-serialization branch from 600631e to bba295d Compare August 26, 2026 09:21

@muranchenhui muranchenhui left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@muranchenhui muranchenhui left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@guslegend0510 guslegend0510 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]:Same-isolation-key concurrent calls race on sandbox state slot

3 participants