Blocking-fresh aft_inspect holds the project execution lane; unrelated agents' mutations queue 25-60s behind it

[iceteaSA]

Blocking-fresh aft_inspect holds a project's execution lane for the full duration of its scan. On a shared root with parallel agents, every unrelated mutation queues behind it. Observed on v0.51.2 with three concurrent implementer sessions on one project.

This is not the v0.51.1 wedge — the deadlines work, nothing hangs forever. It's the cost of the design surfacing under concurrency.

Evidence

Log timestamps are completion times, so a queued call and the inspect that blocked it are logged at the same instant. That pairing is the signature:

T20:02:36  inspect      exec=64.3s              [oa-path-alias-impl]
T20:02:36  apply_patch  queue=24.7s  exec=0.0s  [oa-ws-no-replay-impl]   <- released when inspect ended

T20:04:41  inspect      exec=63.1s              [oa-path-alias-impl]
T20:04:41  safety       queue=53.8s  exec=0.0s  [oa-ws-no-replay-impl]

T20:06:14  inspect      exec=63.0s              [oa-path-alias-impl]
T20:06:14  apply_patch  queue=59.5s  exec=0.0s  [oa-ws-no-replay-impl]

Three inspects, three blocked calls from a different session on the same root, each released exactly when the inspect completed.

The same apply_patch when no inspect is running:

T20:01:23  apply_patch  queue=0.0s  exec=0.2s
T20:03:08  apply_patch  queue=0.0s  exec=0.1s
T20:03:13  apply_patch  queue=0.0s  exec=0.1s
T20:06:33  apply_patch  queue=0.0s  exec=0.1s

0.1–0.2s normally, 25–60s behind an inspect. The lane is healthy except during inspect.

Aggregate over an 8-minute window, 9 sessions:

inspect      n=3   total=190.4s   exec=190.4s   queue=0.0s
apply_patch  n=10  total= 85.3s   exec=  1.0s   queue=84.2s
safety       n=1   total= 53.8s   exec=  0.0s   queue=53.8s

apply_patch spent 84 of its 85 seconds waiting.

Why it's worse than the numbers suggest

BLOCKING_TIER2_PHASE_TIMEOUT is 120s (commands/inspect.rs:24), per phase. The 63s scans above are well inside the bound — a legitimately slow root can hold the lane substantially longer, and the bound is what keeps it finite rather than what keeps it short.

The blast radius is also wider than the caller: the blocked calls came from sessions that never invoked inspect. One agent's diagnostic call stalls every other agent on that root.

Parallel implementer subagents on one repo is the workload where this concentrates, and it's exactly the workload the tool encourages — each agent independently running aft_inspect to check its own edits.

What I'm not claiming

Not a regression against v0.51.1: the deadlines hold and calls terminate. Not a report that inspect is too slow in itself — 63s for a full-root verified-fresh scan may be the honest cost. The issue is specifically that the cost is paid by unrelated concurrent callers on a shared lane, silently, as queue time they cannot see or attribute.

Fix directions

1. Don't hold the mutating lane across the LSP/Tier-2 wait. The v0.51.1 fix for the wedge made the wait terminate; the same shape as LspManager mutex is held across the post-edit diagnostics wait (up to 10s), serializing every other LSP operation #216's parked-waiter would let it wait off-lane. Probably the real fix.
2. Coalesce concurrent inspects on one root — three full scans in four minutes from the same session is redundant work regardless of lane behaviour; the later two could join the first's result.
3. Surface the blocking occupant in the queued call's response so an agent stuck for 60s can see it's waiting on someone else's inspect rather than assuming the daemon is broken. Cheapest of the three and useful even if 1 lands.

Environment

v0.51.2 (aa027c6f, one commit above 1d2da675, the extra commit is an unrelated tool), Linux, subc daemon transport, 9 concurrent sessions across ~6 roots.

[iceteaSA]

Correction: this and #243's second mechanism are the same bug, and my fix suggestion here was wrong.

I filed these as siblings — "a running writer holds the lane" (here) vs "a queued writer can't get it" (#243). One half of that is right; the other collapses.

inspect is a SerialLspStatus command (subc/manifest.rs:124-131), so the 63s inspect measured here holds lsp_inflight, which is a term in has_epoch_reader, which gates Lane::Mutating. The 60s+ SerialLspStatus job starving route binds in #243 is this same job, observed from the bind side instead of the caller side. I noted in #243 that the two "compound" because they showed up on the same root in the same window — that was the tell, and I read it as coincidence.

My fix direction #1 above ("the same shape as #216's parked-waiter would let it wait off-lane") is also wrong, for a layer reason: #216 drops the LspManager mutex; the lane is executor state. lsp_inflight is set once at admission (executor/mod.rs:2118) and cleared once in complete_job (executor/mod.rs:1894) — no mid-job release. The #216 shape is already present on the inspect push-wait (context.rs:5959-5977) and the lane stays held straight through it.

Full corrected analysis, revised fix directions, and a new finding — cold LSP server initialization is 30s per server with no aggregate cap, so a single lsp_hover on a file matching three co-servers can hold the lane for 90s — are in #243 (comment).

Suggest closing this as a duplicate of #243 and consolidating there, unless you'd rather keep this one for the caller-visible symptom (agents queueing 25-60s behind inspect) and track the admission mechanism separately. Your call — happy either way, I just don't want two issues implying two fixes.

[alfonso-aft]

Confirmed, and thank you for the queue=/exec= pairing analysis — the completion-time signature is exactly right. Verified against source: the blocking-fresh inspect request releases the LSP manager lock during quiescence waits (so other LSP work proceeds), but it keeps its executor lane slot for the full scan, and mutating-lane admission waits for lsp_inflight to reach zero. Your 25-60s queue times are that admission wait, precisely as you read it.

The fix direction (design under adversarial review now, since it's executor-concurrency-critical): blocking-fresh inspect moves to the deferred-response model the bash orchestration path already uses — the lane job starts the analysis and registers a pending response, then releases the lane; passive waits (LSP quiescence, thread-pool scans) happen off-lane and the main loop delivers the terminal when the analysis completes. Exclusive phases stay as short bounded jobs. That makes a 60s inspect cost parallel sessions milliseconds of lane time instead of the full minute.

Tracking the reader-churn half of the starvation in #243 (distinct mechanism, same design cycle).

[alfonso-aft]

The lane fix is on main (5d4b84a7): subc aft_inspect now routes through the deferred-response seam — the executor lane job does setup only and releases its slot, passive waits (LSP quiescence, pool scans) happen off-lane, and the terminal is delivered when the analysis completes. A 60s inspect now costs parallel sessions milliseconds of lane time instead of the full minute. The storm-shaped regression asserts your exact presentation: a route bind and a mutation from other sessions admit within normal budgets while an inspect runs.

Also shipped with it, per the adversarial review of the design: cancellation re-homed so route abandonment stops a deferred inspect promptly (not just at phase deadlines), per-root single-flight so off-lane inspects can't stack, shutdown emits inspect terminals with leak assertions, a mid-wait-mutation freshness regression (no stale-FRESH), and the falsification instrument from #243 — the census now records reader ages relative to the oldest queued writer plus a counter of reader admissions while a promoted writer waited (expected 0, forever).

Ships in the next patch. Between this, #241's configurable deadline, and #247's scope semantics, the whole inspect-under-concurrency complex from your and mathiasloh's reports is closed.

[alfonso-aft]

Shipped in v0.51.3.