v1.5.0: WAL-streamer systemd unit (#56), LLM helper fixes, dependency bumps by postgresql007 · Pull Request #63 · cybertec-postgresql/pg_hardstorage

postgresql007 · 2026-09-20T12:33:43Z

Supersedes #61 and #62, which are closed in favour of this branch — one review, one release.

Also carries the four Dependabot PRs (#57–#60) as merges, so they close with it.

1. Issue #56 — the WAL streamer had no systemd unit

pg_hardstorage.service and pg_hardstorage@.service both run pg_hardstorage agent, which executes the scheduled backup and retention engine and never
opens a WAL stream (internal/agent contains no reference to one). The
getting-started tutorial, the R1 runbook and all three migration guides told
operators to supervise the streamer with those units.

Following the tutorial therefore gave you periodic base backups and no
continuous archiving, with nothing to indicate the always-on data plane was
absent. The R1 runbook was worse: systemctl stop pg_hardstorage to "stop the
streamer" left the slot held and PostgreSQL still blocked — the exact failure
that runbook exists to resolve.

adds pg_hardstorage-wal-stream@.service, templated on the deployment
(wal stream takes it as a positional, so %i genuinely selects one),
packaged by all four recipes
corrects the tutorial, the R1 runbook and the three migration guides
three runbooks also named pg_hardstorage-agent, a unit no package ever shipped
new tests bind units to the CLI: every unit must appear in every packaging
recipe, every ExecStart must name a real verb, and some unit must run wal stream

2. LLM helper — three defects found against real endpoints

Found by driving llm ask against live reasoning models rather than the mock provider.

A slow model was treated as a hung connection. http.Client.Timeout: 5m is a
deadline on the whole exchange including the body; an SSE client cannot tell a
dead connection from a healthy stream still delivering tokens.

model	one ordinary question	stream healthy?
`deepseek-v4-flash-0731`	265 s	yes — continuous bytes, clean `[DONE]`
`qwen3.8-flash-next`	532 s	yes

Both killed at 300 s and reported as a context deadline — reading as a network
fault rather than our own client hanging up. Every reasoning model was
affected. Connect/TLS/response-header timeouts are now bounded individually,
and the stream is watched for a gap between bytes.

Every question shipped a 150 KB system prompt. hotCommandPaths bakes 38
commands' full --help into every prompt. Its comment estimated "~200–400
tokens" per entry; measured, it is ~1,000 — 38,309 tokens, a 3× underestimate,
which is why the list was allowed to grow. Asking "is there an RPM?" shipped
the flag inventory of restore, forecast and compliance report. Now
budgeted to 16 KB (PG_HARDSTORAGE_LLM_HOT_HELP_BYTES), with the overflow named
so the model calls read_command_help — already registered, already advertised
in that same prompt.

The validator's retry narrated itself into the answer. Replies came back
opening "Now I have the correct flags. Here's the revised answer." The
operator never saw the first attempt, so the answer began by referring to a
conversation that, from their side, never happened.

Metric	Before	After
System prompt	150 KB / 38,413 tok	64 KB / 16,444 tok
"How do I take my first backup?"	764 s, failed	316 s, answered

3. Dependencies (#57–#60)

#57 matters: grpc 1.83.1 landed in #55 to clear an unreachable advisory and
carried a reachable one — GO-2026-6443, server panic via missing authority
or Host headers, reachable through transport.http2Server.HandleStreams.
1.83.2 clears it. Plus three OTel 1.45.0 bumps.

Verification

Gate	Result
`go build` / `go vet` / `gofmt`	pass / pass / clean
`go test ./cmd/... ./internal/... ./compat/...`	pass
`mkdocs build --strict`	clean
`govulncheck -mode=binary`	0 reachable

TestOperatorEnvVarsAreDocumented caught the new env var as undocumented on the
first run — documented rather than worked around.

Four gates that rebuild bin/pg_hardstorage are not yet run, because an LLM
evaluation is driving that binary and rebuilding mid-run would mix two code
versions into one dataset. They are listed with host caveats in
RELEASE-CHECKLIST-v1.5.0.md, which is scaffolding to delete before tagging.

Suggested version: 1.5.0 — new systemd unit, materially different LLM
prompt, new env var. Nothing on disk or on the wire changes.

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.83.1 to 1.83.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.83.1...v1.83.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.83.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/sdk dependency-version: 1.45.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

…tlptracehttp Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp dependency-version: 1.45.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) from 1.44.0 to 1.45.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](open-telemetry/opentelemetry-go@v1.44.0...v1.45.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace dependency-version: 1.45.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

…ang.org/grpc-1.83.2'

…emetry.io/otel/sdk-1.45.0'

…emetry.io/otel/exporters/otlp/otlptrace/otlptracehttp-1.45.0' # Conflicts: # go.mod # go.sum

…emetry.io/otel/exporters/otlp/otlptrace-1.45.0' # Conflicts: # go.mod # go.sum

The packages shipped pg_hardstorage.service and pg_hardstorage@.service — both running `pg_hardstorage agent` — and the documentation told operators to supervise the WAL streamer with them. The agent runs the scheduled backup and retention engine and never opens a WAL stream (grep internal/agent: no reference to it). So an operator who followed the getting-started tutorial ended up with periodic base backups and no continuous archiving, and nothing said so. Adds pg_hardstorage-wal-stream@.service, templated on the deployment because `wal stream` takes it as a positional, packaged by all four recipes. Corrects the tutorial, the R1 runbook (whose `systemctl stop pg_hardstorage` left the slot held and PG still blocked) and the three migration guides, each of which promised a replication slot and then enabled the agent. Three runbooks also named `pg_hardstorage-agent`, a unit no package ever shipped. New tests bind the units to the CLI: every unit must be in every packaging recipe, every ExecStart must name a real verb, and some unit must run `wal stream`. Also merges #57-#60. #57 matters: grpc 1.83.1, merged four days ago in #55 to clear an unreachable advisory, carried a REACHABLE one (GO-2026-6443). 1.83.2 clears it; the binary scans clean again. Closes #56 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

The OpenAI-compatible provider set http.Client.Timeout to 5 minutes, described in a comment as "a backstop for hung connections". It is not one. Client.Timeout is a deadline on the WHOLE exchange including the body, and this is a Server-Sent Events client — it cannot distinguish a dead connection from a healthy stream still delivering tokens. Measured against a deepseek-v4 reasoning endpoint, one ordinary question ("What does pg_hardstorage wal stream do?") streams steadily for 265s at max_tokens=4096; a qwen3.8 endpoint takes 532s. Both are well-behaved: bytes arrive continuously and the server closes with `data: [DONE]`. The client killed them at 300s and reported openai: read stream: context deadline exceeded (Client.Timeout or context cancellation while reading body) which reads as a network fault rather than as our own client hanging up on a working model. Every reasoning model is affected, and the failure is silent in the sense that nothing points at the real cause. The phases where silence genuinely means a fault are now bounded individually — connect and TLS (30s), response headers (5m, since a busy inference server queues before generating) — and the stream itself is watched for a two-minute gap BETWEEN BYTES by a stall reader, which is the condition the old timeout was reaching for. Total duration belongs to the caller's ctx, which is where the operator's Ctrl-C already lives. A stalled stream now says so, instead of surfacing a bare context error. Verified end to end: the query that died at 300s now returns rc=0 in 265s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

Two defects found by driving `llm ask` against real reasoning endpoints (deepseek-v4, qwen3.8) rather than the mock provider. 1. Every question shipped a 150 KB system prompt. hotCommandPaths bakes the full --help of 38 commands into the prompt at bootstrap. Its comment estimated "~200-400 tokens" per entry; measured against the live binary it is ~1,000 — 38,309 tokens in total, so the list was kept "tight" against a budget understating the cost by 3x. Asking "is there an RPM?" shipped the full flag inventory of restore, forecast and compliance report. The cost is not money, it is latency: 38k tokens of prefill on a reasoning endpoint is minutes of silence before the first token, which is indistinguishable from a hung client. The block is now budgeted to 16 KB (PG_HARDSTORAGE_LLM_HOT_HELP_BYTES, 0 disables), entries render in priority order, and the overflow is named so the model knows to call read_command_help rather than guess. That tool was already registered and already advertised in the prompt. Measured: 150 KB -> 64 KB, 38,413 -> 16,444 tokens. "How do I take my first backup?" went from 764s-and-failing to 316s-and-answering. 2. The validator's retry narrated itself into the answer. When a reply names a bad flag, the session asks the model to revise. Nothing told it to keep that internal. Real answers came back opening "Now I have the correct flags. Here's the revised answer." and "Correct — `wal preflight` takes only --pg-connection. Revised:". The operator never saw the first attempt or the validator's complaint, so the answer simply begins by referring to a conversation that, from where they are standing, did not happen. The retry prompt now says to reply with the corrected answer only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

internal/cli's own docs_surface guard caught PG_HARDSTORAGE_LLM_HOT_HELP_BYTES as read by production code but absent from docs/ — the check exists because an undocumented env var is a setting no operator can find. Also folds the two LLM fixes and the prompt-size change into the Unreleased changelog, which previously carried only the systemd and dependency work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

Records what has been verified on the branch and what still needs a run before tagging — specifically the four gates that rebuild bin/pg_hardstorage, which were deferred because an LLM evaluation run was driving that binary. Scaffolding; delete before tagging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

The 10h soak recorded one fault_apply_failed in 1059 faults: the kernel refused a 32 MiB cgroup cap it could not reclaim to, with swap disabled for the container. The cell was unharmed and the product is not at fault — it is testkit triage noise, filed as #64. Recorded here so whoever reads the soak report before tagging does not have to re-derive it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

The stall watchdog added earlier was calibrated against an IDLE endpoint, where the largest mid-stream gap was 0.7s. Two minutes looked generous. Against a real self-hosted endpoint serving more than one request it was catastrophic: a 194-question run lost 192 answers. Raising it to 15 minutes still lost 6 of 8. Measuring instead of guessing again, with 8 concurrent requests: req headers 1st byte max gap total 5 0.1s 1.0s 0.9s 272s 0 0.1s 273.1s 273.0s 464s 3 0.1s 873.7s 873.6s 1142s Headers come back in 0.1s for every request — the server accepts immediately and then QUEUES. A request can sit silent for 14+ minutes waiting its turn, and that grows with concurrency. But max gap equals first byte in every row: once a request starts generating, tokens arrive continuously and gaps are sub-second. So the long silence is entirely the queue, and it is legitimate. One budget cannot tell it from a dead socket, which is why raising the number did not work and never would have. The two are now separate: PG_HARDSTORAGE_LLM_FIRST_BYTE_TIMEOUT 45m covers the queue wait PG_HARDSTORAGE_LLM_STALL_TIMEOUT 5m after the first byte The error names which budget fired, and says the request may be queued rather than implying a network fault. This is not a test-harness concern. Any on-prem vLLM serving more than one user was hitting it, and a configurable --endpoint exists for exactly that deployment. Re-tested under the condition that failed: 8/8 answered, up from 2/8. The two slowest took 1148s and 1267s and returned real answers citing 5 and 7 genuine commands — both would have been discarded before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

stallReader decides whether a silent stream is a server that has gone away or one that is merely slow, and it had no tests at all. Both ways of getting that wrong have already shipped: a 2-minute budget calibrated on an idle endpoint discarded 192 of 194 answers from a busy one, and a budget with no upper bound hangs the command on a dead socket. Four tests now pin the split — the first-byte budget governs the queue wait, the stall budget takes over once bytes flow, a steadily streaming response is never cut off however long it runs, and the timer may fire while a Read is in flight. Two things worth recording, because both were mistakes this commit had to correct rather than hypotheticals: The first draft of the concurrency test called Read from four goroutines and the race detector duly reported a race — inside strings.Reader, which is not safe for concurrent use. Nothing in the provider reads concurrently (bufio.Scanner pulls from one goroutine), so the scenario could not occur and the finding was noise. The test now models the only concurrency that exists: the timer's callback firing under an in-flight Read. And the first draft of the first-byte test asserted nothing. It inspected the reader without ever calling Read, and the budget is chosen inside Read — so it passed with the fix reverted. Verified by reverting: it now fails with the exact message describing the bug. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

The prompt lists the assistant's tools next to the real command catalog, and describes them in CLI terms ("Run `pg_hardstorage doctor` and return its structured report"). That says what a tool does for us, but leaves the tool NAME looking like part of the command surface. Models duly conflated the two. Across a 194-question run, three answers told the operator to run pg_hardstorage read_command_help deployment add inside a copy-pasteable shell block. That command does not exist and cannot: read_command_help is ours, not theirs. An operator following it gets "unknown command" while attempting the very thing the answer was explaining. Budgeting the hot-command help block made this likelier rather than less — the model is now told more often to reach for read_command_help, so the name appears more often in its context. Naming the boundary is what keeps that change safe. The tools section now opens by stating that these are the assistant's tools, that the operator cannot call them, and that they must never appear in a shown command — naming the one that actually leaked, so the instruction is concrete rather than abstract. Two tests: the preamble must keep making the claim (reword freely, don't drop it), and a tool name in a shell fence must reach the command validator, since nothing else can catch it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

`--repo` is on 101 of 208 command paths — 48%, a coin flip — and the command catalog in the same prompt deliberately shows verbs without flags. So the model has nothing to reason from and guesses. Across a 194-question evaluation against three models it was the most-invented flag by a wide margin: 42, 46 and 69 occurrences, bolted onto doctor, deployment list, kms inspect, wal preflight, schedule and audit. Confirmed against the live binary — every one of those command lines dies immediately with `unknown flag: --repo`. The cheatsheet already listed commands that DO take it. Positive examples alone are what invites the generalisation: the model sees `repair scrub --repo` and concludes the flag is universal. This adds the half that was missing — the common commands that take none, with the flags they take instead, and the reason (they resolve the repository through the deployment in pg_hardstorage.yaml). Cost is ~370 tokens against a 16.4k prompt. A list of facts about the CLI rots when the CLI moves, and a rotted entry is worse than no entry: the model would be told something false by the thing meant to stop it inventing. So the test asks the built binary whether each claim still holds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RmkgG1CUNYTkZES1M4BkkN

dependabot Bot and others added 15 commits September 10, 2026 09:49

Merge remote-tracking branch 'origin/dependabot/go_modules/google.gol…

3870cd2

…ang.org/grpc-1.83.2'

Merge remote-tracking branch 'origin/dependabot/go_modules/go.opentel…

843cd25

…emetry.io/otel/sdk-1.45.0'

Merge remote-tracking branch 'origin/dependabot/go_modules/go.opentel…

9bd1c86

…emetry.io/otel/exporters/otlp/otlptrace/otlptracehttp-1.45.0' # Conflicts: # go.mod # go.sum

Merge remote-tracking branch 'origin/dependabot/go_modules/go.opentel…

eaccc61

…emetry.io/otel/exporters/otlp/otlptrace-1.45.0' # Conflicts: # go.mod # go.sum

Merge branch 'fix/56-wal-stream-systemd-unit' into integration/v1.5.0

ae98143

Merge branch 'fix/llm-streaming-timeout' into integration/v1.5.0

8800c74

This was referenced Sep 20, 2026

fix(systemd): ship a unit for the WAL streamer (#56) + dependency bumps #61

Closed

fix(llm): a slow model is not a hung connection (SSE stream timeout) #62

Closed

testkit: cgroup_squeeze records a kernel limit refusal as fault_apply_failed #64

Open

postgresql007 and others added 5 commits September 20, 2026 16:48

Navigation Menu

v1.5.0: WAL-streamer systemd unit (#56), LLM helper fixes, dependency bumps - #63

v1.5.0: WAL-streamer systemd unit (#56), LLM helper fixes, dependency bumps#63
postgresql007 wants to merge 20 commits into
mainfrom
integration/v1.5.0

postgresql007 commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

postgresql007 commented Sep 20, 2026

1. Issue #56 — the WAL streamer had no systemd unit

2. LLM helper — three defects found against real endpoints

3. Dependencies (#57–#60)

Verification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant