fix(agent): close two run-teardown races in ReusableReadableStream and async drain - #126
abhinav-pola wants to merge 3 commits into
Conversation
ReusableReadableStream.createConsumer() during cancel()'s await of the
source reader registered a consumer at a watermark the second backlog
sweep then cleared, so its first read threw the buffer invariant error.
cancel() now marks the stream cancelled synchronously and later
consumers are done immediately.
handleRunEndAsyncTasks('drain') only called dropSettledTasks() when work
was still in flight, so a task that settled during the last drain turn
was never harvested and stayed persisted as working. The final drop now
runs unconditionally.
Adds specs/tla with the models and TLC configs that found both.
|
I'll fix CI failures and address comments from users with write access that start with 'DevinAI' or '@devin'.
Original prompt from Abhinav
|
…el() cancel() before the first consumer never acquired a reader, and the cancelled flag now stops later consumers from starting the pump, so the source stream would stay open. Cancel it directly in that case. Also extract registerConsumer() so createConsumer() stays under the structural gate's complexity limit.
Summary
Fixes two reachable run-teardown races. Each has a Vitest reproduction that fails on
mainand passes here.1.
ReusableReadableStream.cancel()vs. a latecreateConsumer()(active-consumers replay)cancel()is two-phase: drop backlog,await sourceReader.cancel(), drop backlog again. A chunk the pump already read lands in the buffer during the await and the second drop advancestrimOffsetpast it and clears the slot. A consumer created during the await is registered at the pre-drop watermark, so its firstnext()hits the cleared slot and throwsReusableReadableStream buffer invariant violated: consumed slot was cleared.export class ReusableReadableStream<T> { + private cancelled = false; createConsumer() { const consumerId = this.nextConsumerId++; - this.consumers.set(consumerId, {...}); startPump(); + if (!this.cancelled) { this.consumers.set(consumerId, {...}); startPump(); } // iterator: a consumer not in the map is done immediately } async cancel() { + this.cancelled = true; ... if (this.sourceReader) await this.cancelSourceReader(this.sourceReader); + else if (!this.pumpStarted) await this.cancelUnstartedSource();The
cancelUnstartedSource()branch coverscancel()before any consumer exists: with the terminal flag, no later consumer can start the pump, so the wrapper cancels the unlocked source itself.Reachable from
ModelResult: anygetFullResponsesStream()/getUiStream()call racing a run-level abort orcancel().2.
onRunEnd: 'drain'loses a settlement that lands during the last drain turnif (registry.hasInFlight()) { registry.abortAll('Async tool drain budget exhausted at run end'); - await this.dropSettledTasks(); } + await this.dropSettledTasks();When the loop exits by
maxDrainTurns(or the loop-topsettledQueuecheck misses a settlement that arrives during the following model turn), the task is no longer in flight, so the old conditional skipped the final drop. The settlement stayed insettledQueue, notool.async_settledevent was broadcast, and the persisted task record remainedworking. Now it is broadcast withdelivery: 'dropped'and persisted as terminal, matching the documented drain contract.Verification
pnpm run lint,pnpm run typecheck,pnpm run buildpass.pnpm run test: agent 1352 tests pass, mcp 12 pass.sentrux gatereports no degradation (createConsumerkept under cc 15 by extractingregisterConsumer).src/lib/reusable-stream.tsandsrc/lib/model-result.tsreverted tomain, the three new tests are the only failures.e2e-testsfailure (toolStreamdeltas,BadRequestResponseErrorfrom the live provider) also fails onmain's scheduled run https://github.com/OpenRouterTeam/typescript-agent/actions/runs/35694019034.Searched open PRs; #122 and #117 touch
model-result.tsbut not these paths.Link to Devin session: https://openrouter.devinenterprise.com/sessions/751c0e35dd354e94ba0d8889f6f1cefb
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/751c0e35dd354e94ba0d8889f6f1cefb?variant=devin
Requested by: @abhinav-pola