fix(llm): a slow model is not a hung connection (SSE stream timeout) - #62
Closed
postgresql007 wants to merge 2 commits into
Closed
postgresql007 wants to merge 2 commits into
postgresql007 wants to merge 2 commits into
Conversation
The OpenAI-compatible provider set http.Client.Timeout to 5 minutes,
described in a comment as "a backstop for hung connections". It is not
one. Client.Timeout is a deadline on the WHOLE exchange including the
body, and this is a Server-Sent Events client — it cannot distinguish
a dead connection from a healthy stream still delivering tokens.
Measured against a deepseek-v4 reasoning endpoint, one ordinary
question ("What does pg_hardstorage wal stream do?") streams steadily
for 265s at max_tokens=4096; a qwen3.8 endpoint takes 532s. Both are
well-behaved: bytes arrive continuously and the server closes with
`data: [DONE]`. The client killed them at 300s and reported
openai: read stream: context deadline exceeded
(Client.Timeout or context cancellation while reading body)
which reads as a network fault rather than as our own client hanging
up on a working model. Every reasoning model is affected, and the
failure is silent in the sense that nothing points at the real cause.
The phases where silence genuinely means a fault are now bounded
individually — connect and TLS (30s), response headers (5m, since a
busy inference server queues before generating) — and the stream
itself is watched for a two-minute gap BETWEEN BYTES by a stall
reader, which is the condition the old timeout was reaching for.
Total duration belongs to the caller's ctx, which is where the
operator's Ctrl-C already lives.
A stalled stream now says so, instead of surfacing a bare context
error.
Verified end to end: the query that died at 300s now returns rc=0 in
265s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN
Two defects found by driving `llm ask` against real reasoning endpoints (deepseek-v4, qwen3.8) rather than the mock provider. 1. Every question shipped a 150 KB system prompt. hotCommandPaths bakes the full --help of 38 commands into the prompt at bootstrap. Its comment estimated "~200-400 tokens" per entry; measured against the live binary it is ~1,000 — 38,309 tokens in total, so the list was kept "tight" against a budget understating the cost by 3x. Asking "is there an RPM?" shipped the full flag inventory of restore, forecast and compliance report. The cost is not money, it is latency: 38k tokens of prefill on a reasoning endpoint is minutes of silence before the first token, which is indistinguishable from a hung client. The block is now budgeted to 16 KB (PG_HARDSTORAGE_LLM_HOT_HELP_BYTES, 0 disables), entries render in priority order, and the overflow is named so the model knows to call read_command_help rather than guess. That tool was already registered and already advertised in the prompt. Measured: 150 KB -> 64 KB, 38,413 -> 16,444 tokens. "How do I take my first backup?" went from 764s-and-failing to 316s-and-answering. 2. The validator's retry narrated itself into the answer. When a reply names a bad flag, the session asks the model to revise. Nothing told it to keep that internal. Real answers came back opening "Now I have the correct flags. Here's the revised answer." and "Correct — `wal preflight` takes only --pg-connection. Revised:". The operator never saw the first attempt or the validator's complaint, so the answer simply begins by referring to a conversation that, from where they are standing, did not happen. The retry prompt now says to reply with the corrected answer only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN
This was referenced Sep 20, 2026
Contributor
Author
|
Superseded by #63. Both commits on this branch are contained in Consolidating means one review and one release. The branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while standing up the LLM query harness against the DGX endpoints.
A slow model is not a hung connection
The OpenAI-compatible provider set
http.Client.Timeout: 5 * time.Minute, with acomment calling it "a backstop for hung connections". It is not one.
Client.Timeoutis a deadline on the whole exchange including the body, andthis is an SSE client — it cannot tell a dead connection from a healthy stream
still delivering tokens.
Measured against the live endpoints:
max_tokens=4096deepseek-v4-flash-0731data: [DONE]qwen3.8-flash-nextBoth were killed at 300 s and reported as:
which reads as a network fault rather than our own client hanging up on a
working model. Every reasoning model is affected — and since reasoning
models are exactly what operators point this at, the helper was unusable
against them for any non-trivial question.
The fix
Bound the phases where silence genuinely means a fault, and leave generation
to the caller's ctx:
stallReaderthat resets onevery
ReadThat last one is the condition the old timeout was reaching for: a server that
stops sending without closing. A stalled stream now says so, instead of
surfacing a bare context error.
Verification
The query that died at 300 s now returns
rc=0in 265 s, with a realanswer. Provider and
internal/llm/...unit tests pass.