Skip to content

fix(llm): a slow model is not a hung connection (SSE stream timeout) - #62

Closed
postgresql007 wants to merge 2 commits into
mainfrom
fix/llm-streaming-timeout
Closed

postgresql007 wants to merge 2 commits into
mainfrom
fix/llm-streaming-timeout

Conversation

@postgresql007

Copy link
Copy Markdown
Contributor

Found while standing up the LLM query harness against the DGX endpoints.

A slow model is not a hung connection

The OpenAI-compatible provider set http.Client.Timeout: 5 * time.Minute, with a
comment calling it "a backstop for hung connections". It is not one.
Client.Timeout is a deadline on the whole exchange including the body, and
this is an SSE client — it cannot tell a dead connection from a healthy stream
still delivering tokens.

Measured against the live endpoints:

model one ordinary question, max_tokens=4096 stream healthy?
deepseek-v4-flash-0731 265 s yes — continuous bytes, clean data: [DONE]
qwen3.8-flash-next 532 s yes — same

Both were killed at 300 s and reported as:

openai: read stream: context deadline exceeded
  (Client.Timeout or context cancellation while reading body)

which reads as a network fault rather than our own client hanging up on a
working model. Every reasoning model is affected — and since reasoning
models are exactly what operators point this at, the helper was unusable
against them for any non-trivial question.

The fix

Bound the phases where silence genuinely means a fault, and leave generation
to the caller's ctx:

  • connect + TLS handshake — 30 s
  • response headers — 5 min (a busy inference server queues before generating)
  • gap between bytes mid-stream — 2 min, via a stallReader that resets on
    every Read

That last one is the condition the old timeout was reaching for: a server that
stops sending without closing. A stalled stream now says so, instead of
surfacing a bare context error.

Verification

The query that died at 300 s now returns rc=0 in 265 s, with a real
answer. Provider and internal/llm/... unit tests pass.

postgresql007 and others added 2 commits September 20, 2026 13:07
The OpenAI-compatible provider set http.Client.Timeout to 5 minutes,
described in a comment as "a backstop for hung connections". It is not
one. Client.Timeout is a deadline on the WHOLE exchange including the
body, and this is a Server-Sent Events client — it cannot distinguish
a dead connection from a healthy stream still delivering tokens.

Measured against a deepseek-v4 reasoning endpoint, one ordinary
question ("What does pg_hardstorage wal stream do?") streams steadily
for 265s at max_tokens=4096; a qwen3.8 endpoint takes 532s. Both are
well-behaved: bytes arrive continuously and the server closes with
`data: [DONE]`. The client killed them at 300s and reported

    openai: read stream: context deadline exceeded
      (Client.Timeout or context cancellation while reading body)

which reads as a network fault rather than as our own client hanging
up on a working model. Every reasoning model is affected, and the
failure is silent in the sense that nothing points at the real cause.

The phases where silence genuinely means a fault are now bounded
individually — connect and TLS (30s), response headers (5m, since a
busy inference server queues before generating) — and the stream
itself is watched for a two-minute gap BETWEEN BYTES by a stall
reader, which is the condition the old timeout was reaching for.
Total duration belongs to the caller's ctx, which is where the
operator's Ctrl-C already lives.

A stalled stream now says so, instead of surfacing a bare context
error.

Verified end to end: the query that died at 300s now returns rc=0 in
265s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN
Two defects found by driving `llm ask` against real reasoning
endpoints (deepseek-v4, qwen3.8) rather than the mock provider.

1. Every question shipped a 150 KB system prompt.

hotCommandPaths bakes the full --help of 38 commands into the prompt
at bootstrap. Its comment estimated "~200-400 tokens" per entry;
measured against the live binary it is ~1,000 — 38,309 tokens in
total, so the list was kept "tight" against a budget understating the
cost by 3x. Asking "is there an RPM?" shipped the full flag inventory
of restore, forecast and compliance report.

The cost is not money, it is latency: 38k tokens of prefill on a
reasoning endpoint is minutes of silence before the first token, which
is indistinguishable from a hung client. The block is now budgeted to
16 KB (PG_HARDSTORAGE_LLM_HOT_HELP_BYTES, 0 disables), entries render
in priority order, and the overflow is named so the model knows to
call read_command_help rather than guess. That tool was already
registered and already advertised in the prompt.

Measured: 150 KB -> 64 KB, 38,413 -> 16,444 tokens. "How do I take my
first backup?" went from 764s-and-failing to 316s-and-answering.

2. The validator's retry narrated itself into the answer.

When a reply names a bad flag, the session asks the model to revise.
Nothing told it to keep that internal. Real answers came back opening
"Now I have the correct flags. Here's the revised answer." and
"Correct — `wal preflight` takes only --pg-connection. Revised:".

The operator never saw the first attempt or the validator's complaint,
so the answer simply begins by referring to a conversation that, from
where they are standing, did not happen. The retry prompt now says to
reply with the corrected answer only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN
@postgresql007

Copy link
Copy Markdown
Contributor Author

Superseded by #63.

Both commits on this branch are contained in integration/v1.5.0, together with the #56 systemd work and the dependency bumps. #63 also adds documentation for PG_HARDSTORAGE_LLM_HOT_HELP_BYTESTestOperatorEnvVarsAreDocumented failed on the integration branch because the new env var was read by production code and appeared nowhere in docs/, which is exactly what that guard is for.

Consolidating means one review and one release. The branch fix/llm-streaming-timeout is left in place; delete it whenever you like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant