Rebuild an ASR encoder graph when the cached one is too big

[christopherthompson81]

Fixes #617.

On the CPU backend, one long transcription leaves a latency floor behind: every later request, however short, pays roughly the long request's price until the model is unloaded or the process restarts. Reproduced, fixed, and measured below.

What the state actually is

Not a leak, and not a cache in the KV sense — it is the encoder compute graph. ensure_graph() reuses a cached graph whenever it is at least as large as the request:

// src/models/nemotron_asr/encoder.cpp
graph_->input_frames >= input_frames &&

and encode() zero-pads the audio up to that capacity. The masks are then filled to the valid length, so the padding is masked out of the result but not out of the arithmetic. The graph executes at its built size on every call, and conformer self-attention is quadratic in frames.

So the capacity only ever ratchets upward — a larger request rebuilds, nothing ever lowers it — and the longest clip the process has ever seen becomes a floor under every later one. That is why the reporter measured the penalty scaling with the previous request rather than being a constant, and why unload_models clears it instantly: it drops the graph along with the model.

This was already solved once here

parakeet_tdt has the same encoder shape, hit the same bug, and carries the fix with its own measurements:

a 7.4s clip costs 1018 ms on a matched graph and 10928 ms on a 60s-capacity one, while rebuilding costs ~400 ms once (dominated by the 24 positional projections; the allocation itself is ~0.4 ms)

nemotron_asr and hviske_asr never got it. Rather than add a third copy of the constant, the rule moves into engine::modules::asr_graph_capacity_usable() in asr_helpers, and all three encoders call it. Parakeet keeps its measured numbers in a comment but no longer owns the threshold.

A cached graph is reused only while it is no more than 10% larger than the request. The tolerance keeps a stream of clips whose lengths wobble slightly from rebuilding on every call, while capping the wasted compute at roughly the same fraction.

Measured

audiocpp_server, --backend cpu, threads=8, lazy_load, Nemotron 3.5 ASR Streaming 0.6B q8_0. Clips cut from assets/resources/speech.wav to the reporter's 33 s and 262 s at 16 kHz mono. Same machine, same flags, before/after differing only in this patch:

request	before	after
33 s (fresh)	3.9 s	3.9 s
33 s	3.1 s	3.2 s
262 s	31.4 s	29.1 s
33 s (after long)	25.7 s	3.2 s
33 s (after long)	24.9 s	2.9 s
RSS	4377032 KiB	1664236 KiB

An 8.3x penalty, against an 8x ratio between the clip lengths — which is what a graph running at the longer capacity should cost. This box is far faster than the reporter's Pi 5 in absolute terms, but it is the same phenomenon at the same ratio. The long request itself is unchanged (31.4 → 29.1 s is run-to-run noise), so the rebuild is not being paid anywhere visible.

Two corroborations worth noting:

• RSS landed at 4.17 GB against the reporter's ~4.3 GB. Two machines do not leak to the same number by coincidence; the compute buffer is a deterministic function of the graph. The after figure is measured once the process is back on a small graph — peak usage during the 262 s request is necessarily unchanged, since that graph still has to be built once.
• The output is unaffected, and I checked. Transcribing the 33 s clip on a fresh matched graph and again on a 262 s-capacity graph, on the unpatched binary, gives byte-identical text. The masks remove the padded frames from the result exactly; the only thing an oversized graph costs is arithmetic on frames that are then discarded. So [Bug] Server: large fixed latency penalty on every subsequent request after one long request (CPU backend, Nemotron ASR q8_0) #617 is purely a latency and memory defect, and this PR claims no correctness fix. It also settles the tolerance question from the other direction: if an 8x oversize changes nothing, the 1.1x one allowed here changes nothing.

Also in this change

relative_positional_encoding_cache_ is keyed by frame count and unbounded, and each entry is (2 * frames - 1) * hidden floats — tens of MB at conversational lengths. Under the old behaviour few distinct sizes ever occurred, so it stayed small; making rebuilds common would have turned it into a genuine growth path in a long-lived server. It is now bounded in nemotron_asr and parakeet_tdt. This trades nothing for the fix, but it is a second-order consequence of it rather than part of the reported bug.

tests/unittests/test_asr_graph_capacity.cpp covers exact fit, undersized, inside-tolerance, the #617 case (an 8x oversized graph must rebuild), and zero sizes. It needs no weights, so CI runs it. Verified that it fails without the fix.

Scope and limits

• hviske_asr is fixed by inspection, not by measurement. Same code shape and the same reasoning applies, but I only had Nemotron weights on hand; I have not run a before/after on it.
• The 10% tolerance is inherited from Parakeet's measurements, not re-derived for Nemotron. Nemotron builds its own positional projections, so its rebuild cost could differ. The before/after shows the constant works here; I did not measure Nemotron's rebuild cost directly to confirm 10% is the right knee.
• The streaming path is untouched — it was already excluded by the !graph_->streaming guard, and its chunk sizes are fixed.

Interaction with prepare_capacity(). prepare() builds a graph sized from max_input_samples, and encode() then asks for the real frame count. Where those agree — the server's offline path, per request — nothing changes: prepare builds it, encode reuses it, and the measured 3.2 s confirms no build is being paid twice. Where a caller passes a ceiling much larger than the audio, prepare's graph is now discarded on first encode instead of being reused at the wrong size. That is a real cost, but it is the same trade parakeet_tdt already makes, and the alternative is the bug this fixes. prepare_capacity()'s other job — failing early when a request exceeds max_position_embeddings — is unaffected.

Other encoders sharing the shape, deliberately not touched here. Checked while scoping: sense_asr, fun_asr_nano (encoder and adaptor) key their graph cache on exact frame equality, so they never reuse an oversized graph and have no version of this bug. But index_tts2/semantic_encoder, ace_step/condition_encoder, qwen3_tts and vietneu_tts speaker encoders, audio8_tts/codec and fish_dac_codec_runtime all reuse on a bare capacity >= request, and all of them carry attention. They are plausibly the same latent bug on TTS and codec paths. I have not measured any of them and their input sizes are driven by text or tokens rather than audio length, so I would rather flag them than fix them blind in a PR about ASR latency. Happy to open a separate issue if that is useful.

Full suite: 36/36.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QCCx4pMMex72qrNomPh5yL