Route-bind configure starves on the reader gate: 18-204s ack_ready waits, two distinct mechanisms

[iceteaSA]

Under parallel-agent load on one root, configure (route bind) sits in the Mutating lane waiting for read_inflight/lsp_inflight to reach zero, and never gets in. Observed waits of 18s → 68s → 82s → 96s → 160s → 204s on a single root. Every phase that does real work reports 0ms:

configure_phase_timings=[canonicalize=0ms, worktree_probe=0ms, config_resolve=0ms, ack_ready=95989ms]
blockers=[queued_behind_configure(1), waiting_on_readers]

No slow project walk, no gitignore scan, no DB open. All of it is admission wait.

Because the daemon's route-bind relay times out well before that, new binds fail while already-bound routes keep working — so it presents as "aft is broken for new sessions" while existing sessions are fine.

Two distinct mechanisms, not one

The blocker census distinguishes them, and the split matters because a single fix only addresses one:

16x  waiting_on_readers            <- reader churn: many short reads, none individually old
 3x  waiting_on_readers_stuck(...) <- one long-lived reader, named

(a) Reader-churn starvation — 16 occurrences, the 68–96s band. Admission is reader-preferring with no fairness (executor/mod.rs:2078,2090):

Lane::PureRead => actor.read_inflight < config.read_cap && actor_has_capacity,
Lane::Mutating => !has_epoch_reader && actor_has_capacity,

A new PureRead is admitted whenever the cap allows. Nothing consults the Mutating queue, so a continuous stream of short reads keeps read_inflight above zero indefinitely and the queued writer is lapped forever. Self-reinforcing: more parallel sessions → denser reads → longer starvation.

A write barrier already exists but is scoped one class too narrowly (executor/mod.rs:~1200):

fn higher_priority_writer_barrier_blocks(&self, job_class: JobClass) -> bool {
    matches!(job_class, JobClass::Maintenance)
        && !self.interactive.queue(Lane::Mutating).is_empty()
}

A queued interactive writer blocks maintenance from starting, but not interactive reads. The concept is in the codebase; it just doesn't cover the case that starves.

(b) A single long reader on the LSP lane — 3 occurrences, and they are the WORST waits (160s, 204s). All three name the same lane:

waiting_on_readers_stuck(job=subc-56-84  command=executor::Interactive::SerialLspStatus
                         lane=SerialLspStatus root=/…/openai-auth age_ms=66289)
waiting_on_readers_stuck(job=subc-82-389 command=executor::Interactive::SerialLspStatus
                         lane=SerialLspStatus root=/…/openai-auth age_ms=62200)

has_epoch_reader includes lsp_inflight, so one 60s+ LSP-status job gates the Mutating lane on its own. Write-preferring admission does not fix this — with a single reader holding for 66 seconds, the writer waits 66 seconds no matter how it is prioritised.

Why the split is the point

The obvious fix (make admission write-preferring or fair) resolves (a) and leaves (b) — which produced the two longest stalls observed. Worth treating as two items:

1. Fair/write-preferring admission: extend the existing barrier so a queued interactive Mutating job blocks new PureRead admissions. Needs a bound so a permanent write barrier can't stall reads indefinitely, but the mechanism is already there for maintenance.
2. Bound or off-lane the long LSP-status reader: a 60s+ SerialLspStatus job holding lsp_inflight blocks every route bind on the root. Either bound it, or let it wait without holding the epoch-reader gate — the same shape as LspManager mutex is held across the post-edit diagnostics wait (up to 10s), serializing every other LSP operation #216's parked-waiter fix.

Relationship to #242

Sibling, not duplicate. #242 is a running writer (blocking-fresh inspect) holding the lane while unrelated mutations queue. This is a queued writer that can't get the lane because readers never yield. Same admission design, opposite ends; different fixes.

The two also compound: the 60s+ SerialLspStatus reader here and the 63s inspect in #242 are on the same root, in the same window.

Environment and attribution

v0.51.2 (aa027c6f), Linux, subc daemon transport, ~9 concurrent sessions across ~6 roots, several parallel implementer agents on /…/openai-auth invalidating files every 1-2s. Not resource pressure: 38G RAM available, load 3.04, daemon at 2.6G.

Starvation mechanism (a) was diagnosed independently by a peer operator working the daemon side, from the ack_ready/blockers timings; I verified it against the admission code and added the (b) split from the stuck-reader census.

[iceteaSA]

Still occurring a day later, on a second box, with queue depth growing. Also a sampling caveat on the original report that I'd rather state myself than have someone discover.

Fresh occurrences, different machine

Six events in three minutes today, 15:28–15:31Z, under parallel-agent load:

blockers=[queued_behind_configure(2), waiting_on_readers]   x4
blockers=[queued_behind_configure(1), waiting_on_readers]   x2
ack_ready = 35914ms · 47987ms · 69615ms · 73632ms · 81666ms · 85697ms

queued_behind_configure(2) is new. Yesterday's sample was uniformly (1); four of six today show a second configure already stacked behind the first. Configures are now queueing behind each other, not just behind readers — so the backlog compounds rather than draining between binds.

The 36–86s band is squarely mechanism (a), reader-churn starvation. Consistent with the original split.

Sampling caveat on mechanism (b) — please weight it accordingly

My "all three stuck censuses name SerialLspStatus" claim is n=3, one box, one project root. I tried to get an independent counterexample and could not, for a structural reason worth recording:

The labelled census only exists in v0.51.1+. waiting_on_readers_stuck and the lane=/age_ms= fields arrived in 5410771b ("bound and cancel blocking inspect"), first tagged v0.51.1 on 2026-08-17. Every build before that emits only the bare waiting_on_readers token. The peer operator I asked has 6 events in their retention and zero lane= strings anywhere, ever — their instrument has one bit where mine has two, so their null is not corroboration and I'm not counting it as such.

Practical consequence: nobody who hit this before Aug 17 can produce a labelled sample retroactively, so mechanism (b)'s evidence base will stay thin for a while. If you have labelled captures from other roots, they'd discriminate quickly — specifically, any waiting_on_readers_stuck naming a lane other than SerialLspStatus would break my framing, and I'd rather that surface before a fix is built around it than after.

What I'm confident in regardless of sample size, because it's read off the code rather than the logs: has_epoch_reader includes lsp_inflight, so whatever occupies SerialLspStatus for 60s gates the Mutating lane on its own, and write-preferring admission does not address that path. The lane identity is n=3; the blocking relationship is structural.

Operator mitigation is narrower than I implied

I suggested "reduce concurrent sessions per root" as the interim lever. Worth knowing it is currently the only one: on the daemon side, DEFAULT_ROUTE_BIND_RELAY_TIMEOUT (12s) has a setter whose sole caller is test code — hardcoded in production with no config path. So an operator hitting this cannot buy headroom by raising the bind deadline while a fix lands; they can only reduce load. That's a separate-repo defect and is being filed there, but it bears on how much urgency this one carries.

[iceteaSA]

Correcting myself on three points, one of which invalidates the fix I proposed for mechanism (b). I ran a source investigation into what actually occupies SerialLspStatus for 60s and verified each claim below against the tree at 1d2da675.

1. #242 and mechanism (b) here are the SAME bug, not siblings

inspect is a SerialLspStatus command (subc/manifest.rs:124-131):

"status"
| "inspect"
| "lsp_diagnostics"
| "lsp_inspect"
| "lsp_hover"
| "lsp_goto_definition"
| "lsp_find_references"
| "lsp_prepare_rename" => Some(Lane::SerialLspStatus),

So the 63s blocking inspect I reported in #242 and the 60s+ SerialLspStatus job here are one job seen from two ends: inspect is admitted to SerialLspStatus, holds lsp_inflight, and lsp_inflight is a term in has_epoch_reader, which gates Lane::Mutating.

I wrote in #243 that the two "compound" because they appeared on the same root in the same window. That was the tell and I misread it as coincidence. Treat #242 as the caller-side view and this as the bind-side view of one mechanism.

2. The #216 parked-waiter fix I proposed does not work — wrong layer

I suggested applying "the same shape as #216's parked-waiter fix" to release the lane during the passive wait. That fix operates on the LspManager mutex; the lane is held by executor state, and dropping one does nothing to the other.

lsp_inflight has exactly one set site and one clear site:

executor/mod.rs:2118   actor.lsp_inflight = true;    // at admission
executor/mod.rs:1894   actor.lsp_inflight = false;   // inside fn complete_job

There is no mid-job release path. The #216 shape is in fact already present on the inspect push-wait and the post-edit path (context.rs:5959-5977 — register waiter, drop(lsp), wait lock-free, reacquire), and the lane stays occupied throughout it anyway.

Releasing the lane during a passive wait needs executor parking/continuation, or splitting the operation into separate jobs — a materially larger change than I implied. The pull-diagnostics path additionally holds LspManager across send_request_with_timeout (lsp/manager.rs:2067-2084), so that one would also need a split-phase request API.

Apologies for pointing at a fix that addresses a different layer.

3. lsp_quiescence_timeout is a deadline label, not evidence of an RA wait

I cited that identifier as evidence the lane was blocked on a language-server wait. It is emitted when the generic 120s diagnostics phase deadline expires (inspect/diagnostics_category.rs:250-260), regardless of what consumed the time. Rust-analyzer quiescence is tracked, not awaited — lsp/client.rs:488-503 only marks Rust diagnostics provisional until the server reports quiescent, and lsp/manager.rs:2457-2477 promotes them on the event. The name reads as a diagnosis and isn't one; worth renaming on its own merits.

4. New: a cheap-looking interactive command can hold the lane past 60s, unbounded in aggregate

This is the part I'd weight highest, because it is worse than inspect.

cold server initialize     30s per server   (lsp/client.rs:25-28, manager.rs:2479-2529)
per-file pull              10s              (manager.rs:1780-1805)
workspace pull             10s              (manager.rs:1924-2031)
interactive LSP request     8s              (client.rs:522-618)
inspect diagnostics phase 120s              (inspect/diagnostics_category.rs:18)
aggregate SerialLspStatus  NONE

Only inspect has a phase bound. Everything else is bounded per-server and per-request, and initialization loops all matching servers (manager.rs:733-803). A single file can match several: your own registry tests cover TypeScript + Biome + Oxlint as co-servers on one TS file (lsp/registry.rs:961-967). Three cold servers is 90s of initialization, held on the lane, from one lsp_hover.

That likely matters for the observed root specifically — /…/openai-auth is TypeScript-heavy, so it is exactly the multi-co-server case.

Revised fix directions

1. Aggregate cap on SerialLspStatus — the cheapest real mitigation. Bounds every occupant including the multi-server-init path, which currently has no ceiling at all.
2. Don't hold the lane across cold server initialization. Initialization is startup work, not the request; moving it off-lane (or making it a separate job) removes the unbounded case without touching the diagnostics design.
3. Fair/write-preferring admission — still the right fix for mechanism (a), the reader-churn band, and unaffected by the above.
4. Renaming lsp_quiescence_timeout to reflect that it is a phase deadline.

Method note: findings produced by a read-only source investigation; I verified the lane mapping and the lsp_inflight set/clear sites myself before posting. The remaining gap is runtime — source cannot tell us which job produced the observed census or how many servers matched that root. Phase traces separating startup / pull / push-fallback time would settle it, and would also confirm or kill point 4.

[iceteaSA]

Field data from a second root closes the attribution gap: the 60s+ SerialLspStatus holder is aft_inspect, confirmed by job-id linkage rather than inference. Also a clean negative finding on partial writes.

My earlier source investigation could name the lane but not which job occupied it — I flagged that as the open gap. This closes it.

The stuck reader is an inspect, and the ids prove it

census:   blockers=[queued_behind_configure(2),
            waiting_on_readers_stuck(job=subc-832-157
              command=executor::Interactive::SerialLspStatus lane=SerialLspStatus
              root=/home/icetea/.config/opencode age_ms=66024)]

holder:   slow tool_call name=inspect      channel=832 corr=157  total=77187ms exec=77185ms
victim:   slow tool_call name=apply_patch  channel=833 corr=159  total=71245ms queue=71208ms exec=36ms

The linkage is exact, not circumstantial: request ids are minted as format!("subc-{}-{}", frame.header.channel, frame.header.corr) (subc/mod.rs:4258), so subc-832-157 is channel 832 / corr 157 — the same channel and corr the inspect line reports.

So: a 77-second aft_inspect on channel 832 held lsp_inflight; an apply_patch on channel 833 sat in queue 71.2s and executed in 36ms once released; the route bind behind it timed out. That is mechanism (b) end to end, with the holder named.

It also confirms the correction I posted above — inspect is a SerialLspStatus command, so #242 and this are one bug — from runtime rather than from the lane table.

Lane sample is now 4-for-4, from a second root

I asked for a counterexample: any waiting_on_readers_stuck naming a lane other than SerialLspStatus. This sample doesn't break it — two more censuses (age_ms=66024, age_ms=71386), both SerialLspStatus, on /home/icetea/.config/opencode rather than the openai-auth root from my original report.

Different root, different session, same lane. Still a small sample and still one machine, but no longer one project.

One thing it does soften: I suggested the TypeScript-heavy root might matter because of multi-co-server cold-init. A second TS-ish root is consistent with that but doesn't isolate it — inspect's own 120s phase budget is sufficient to explain 77s without invoking co-server init at all. Treat the cold-init ceiling (30s × N servers, no aggregate cap) as a separate unbounded path rather than the explanation for these observations.

Negative finding: apply_patch fails closed under this, no partial writes

Worth recording because the failure mode invites the opposite assumption. When the queued apply_patch timed out, git status on all six target files was clean — zero partial writes. queue=71208ms exec=36ms is the proof: it never entered execution, it aged out in the queue. There is no partial-write defect class here; the starvation costs latency and failed binds, not torn files.

Bearing on the fix

Making the inspect path yield the reader gate — or bounding/interrupting it — removes the trigger for mechanism (b) directly, since inspect is now the observed occupant in 4/4 samples rather than a hypothesised one. That's distinct from the fair-admission fix for mechanism (a), which stands on its own for the reader-churn band.

Operator mitigation confirmed working in the field: stop dispatching edit-heavy work into reader contention, let readers drain, run the edit batch solo.

[alfonso-aft]

Confirmed as two distinct mechanisms, matching your census split:

1. lsp_inflight held by long jobs — dominated by blocking-fresh inspect holding its lane through passive waits; that half is Blocking-fresh aft_inspect holds the project execution lane; unrelated agents' mutations queue 25-60s behind it #242's deferred-response redesign.
2. Reader churn — the writer barrier currently waits for in-flight readers to drain to zero while new readers keep being admitted, so under steady read traffic a writer (and configure/bind is one) can wait unboundedly. Your 18→204s escalation is the signature of that missing write-preference. The fix is standard write-preferring admission: once a writer has waited past its grace window, new reader admission pauses until the writer gets in. The existing 6s reads-first promotion orders the queue but does not gate admission — that's the gap.

Both fixes are in one design cycle now (executor-concurrency changes go through adversarial review here before code). The bind-relay presentation you describe — "new sessions broken, existing sessions fine" — is the same admission wait surfacing through the daemon's bind deadline, and it's the acceptance test for the fix: binds must admit within their relay budget under saturated read traffic.

[iceteaSA]

Before you spend a design cycle on write-preferring admission: I think it already exists, and I think my two-mechanism split — which you confirmed — is wrong. Retracting my own framing here, with the reading that changed my mind.

You wrote: "the writer barrier currently waits for in-flight readers to drain to zero while new readers keep being admitted" and "the existing 6s reads-first promotion orders the queue but does not gate admission — that's the gap."

Reading the admission path, the gate looks present.

The promotion returns a single lane, and admission does not fall through

executor/mod.rs:1467-1483:

fn next_interactive_lane(&self, now: Instant) -> Option<Lane> {
    let starved_writer = self.mutating.iter().any(|job| {
        now.saturating_duration_since(job.queued_at) >= INTERACTIVE_WRITER_PROMOTION_AGE
    });
    if starved_writer {
        // Also stops NEW readers from being admitted on this actor while
        // the promoted writer waits for in-flight readers to drain.
        return Some(Lane::Mutating);
    }
    if !self.pure_reads.is_empty() {
        return Some(Lane::PureRead);
    }
    // ...
}

try_admit_actor takes that one lane and, if it isn't runnable, returns None (:2062-2098) — it never tries a different lane in the same pass. So once any mutating job has been queued ≥ INTERACTIVE_WRITER_PROMOTION_AGE (6s, :80), the only lane admission will consider is Mutating, and no new reader is admitted until the writer gets in.

Route binds reach that queue: configure is submitted on Lane::Mutating at subc/mod.rs:3893, so self.mutating sees it and the promotion fires. executor/tests.rs::starved_bind_promotes_over_pure_reads asserts this directly — its comment reads "new reads must stop being admitted while it waits."

If that reading is right, a writer cannot wait 68-204s because new readers keep arriving. It can only wait that long because in-flight readers don't finish — which is mechanism (b), the long-lane-holder.

Why I split them in the first place — the census threshold misled me

I treated waiting_on_readers vs waiting_on_readers_stuck as churn vs single-holder. It isn't. executor/mod.rs:1212-1231:

let stuck_readers = self.running_jobs.values().filter(|job| {
    job.root_id == *root_id
        && matches!(job.lane, Lane::PureRead | Lane::SerialLspStatus)
        && now.saturating_duration_since(job.started_at) >= READER_STUCK_CENSUS_AGE
}).collect::<Vec<_>>();
if stuck_readers.is_empty() {
    blockers.push("waiting_on_readers".to_string());
} else {
    blockers.push(format!("waiting_on_readers_stuck({})", format_running_jobs(&stuck_readers)));
}

READER_STUCK_CENSUS_AGE is 60s (:83). The two strings distinguish "a reader has been running ≥60s" from "readers are in flight, none ≥60s yet" — a naming threshold, not a mechanism. A single reader at 55s renders as the generic string, indistinguishable from churn.

That fully explains my census. The 36-96s band I called "reader churn" is the same long-holder mechanism, just below the naming threshold; the 160-204s cases crossed it and got named. Two buckets of one distribution, cut at 60s.

What this changes

If it holds, the #242 deferred-response redesign fixes both census classes, and the write-preferring admission work is unnecessary — the 6s promotion already does it. That would collapse your design cycle to one change rather than two.

How to falsify me

The reading predicts: no writer should ever wait ≫6s while new readers are being admitted in that window. A single counterexample — a queued mutating job past 6s with a reader started after it — kills it, and your gap statement stands.

I can't produce that from my captures: the census reports in-flight readers but not their start times relative to the writer's enqueue, so I can't tell "arrived after" from "still running." A reader started_at in the stuck census, or a counter of reader admissions while a promoted writer waits, would settle it — and would be worth having as a regression signal for the acceptance test you named regardless of which of us is right.

Stated as a correction rather than a question because I'd rather be wrong in public than have you build against my mis-split. You know this scheduler far better than I do; if the promotion has a hole I'm not seeing, the two-mechanism read stands and I'll say so.

[alfonso-aft]

Your retraction is correct, and I'm withdrawing my "promotion orders the queue but does not gate admission" statement — that was wrong. Verified your reading at source, plus the half you couldn't see:

1. Interactive class — exactly as you traced: next_interactive_lane returns only Mutating once any mutating job passes the 6s promotion age, and try_admit_actor takes that single lane or returns None. No fall-through, no new readers.
2. Maintenance class — stricter than your reading needed: higher_priority_writer_barrier_blocks(Maintenance) refuses ALL maintenance admission the moment any interactive writer is queued — no 6s wait at all. So LSP drains and tier-2 commits can't sneak lsp_inflight/maintenance_commit_inflight holds in either. The gate is complete on both classes.

Your census re-read also survives scrutiny: waiting_on_readers vs _stuck is one distribution cut at the 60s naming threshold, not two mechanisms. Both census classes are the same long-holder story: a writer waits ≤6s of open admission plus the runtime of holders admitted before the gate closed — and on v0.51.2 the dominant long holder is exactly #242's blocking inspect (60-200s of legitimate lsp_inflight). One mechanism, one fix. The design cycle collapses to the deferred-response redesign, as you predicted.

Your falsification instrument ships regardless — it's the missing evidence surface: the census gains each in-flight reader's start-age relative to the oldest queued writer's enqueue, plus a counter of reader admissions while a promoted writer waits (expected always 0; nonzero = admission-gate regression). That settles any future version of this argument in one capture, and doubles as the acceptance probe for the redesign.

"I'd rather be wrong in public than have you build against my mis-split" — this saved a design cycle from building a mechanism that already exists. The correction is the contribution.

[alfonso-aft]

Closing the loop here too: the single-mechanism fix landed on main (5d4b84a7 — #242 has the details), and your falsification instrument shipped exactly as sketched: the bind-blocker census now records each in-flight reader's start relative to the oldest queued writer's enqueue, and a per-actor counter of reader admissions while a promoted writer waited — expected zero always; any nonzero is an admission-gate regression caught by instrumentation instead of a debate.