Conversation
…aph cache nemotron_asr: process_audio_chunk only buffered samples and the whole stream was decoded in finalize(), so 'streaming' produced a burst of deltas only after the audio ended. The decoder already was incremental inside decode_streaming(); this exposes that state machine as begin_stream_decode / decode_stream_chunk / finish_stream_decode and rewires the session to encode + decode each native chunk as it lands (cache-aware encode_stream_chunk, publisher chunk geometry), with a zero-padded final window so the tail is not dropped. granite5asr: the streaming session was a stub (accumulate + offline decode at close). It now runs the publisher's chunked-CTC recipe: every center chunk (granite5asr.center_chunk_sec, default 1 s) is re-encoded together with its left context (granite5asr.left_context_sec, default 2 s) and CTC-greedy decoded continuously, streaming partials per chunk. The window-to-CTC-frame mapping accounts for the encoder's subsample_layers stride (frames advance every hop * stack * 2^|subsample| samples). Session options declared in the model spec. granite5asr encoder: transcribe_features rebuilt the whole 16-block conformer graph per call; graphs are now cached per input shape (small LRU), which removes the per-chunk/per-request build latency. decode_streaming() is now a thin wrapper over the incremental API, so pull-style callers keep their behavior while live callers (server /live) get true per-chunk partials. Verified on the Windows CPU + Vulkan builds: nemotron and granite now stream word-by-word partials through /v1/audio/transcriptions/live (previously zero partials), end-of-turn latency drops accordingly, and offline transcripts are unchanged.
…vance
decode_stream_chunk() iterated the chunk's encoded frames with a for
loop, moving to the next frame after EVERY token. The TDT loop may emit
up to max_symbols_per_step tokens on the same frame (advancing only on
blank or the symbol cap, exactly like decode()/decode_streaming());
advancing per token dropped those extra symbols, which truncated words
('midnight' -> 'night'). Restructured to a while loop that advances the
frame pointer only on blank/force-advance.
Measured at 1 CPU thread on a 6.4 s sample: center 2 s cuts total
streaming encoder compute ~30% (3390 vs 4807 ms, 4 windows vs 7) at
equal transcript quality, but grows the final flush window and adds
~20-25% end-of-turn latency (1256 vs 1011 ms) plus coarser partials.
Left contexts below 2 s duplicate words across window boundaries
('midnight midnight') because the continuous CTC collapse cannot dedupe
re-decoded boundary tokens across different windows. Defaults stay at
center 1 s / left 2 s for latency; document the compute knob.
Contributor
Author
|
work in progress do not review yet. |
Granite's attention is block-local: each context_size (128) frame block attends only within itself, so streaming chunks never needed a left- context window at all — that was a 2.9x compute amplification for nothing. The streaming session now encodes each center chunk exactly once, prepended only with a short (0.3 s) waveform carry that covers the two stride-2 subsample blocks' convolution reach, the avg-pool pair alignment, and the frontend's reflect padding; tokens fully inside the carry region are skipped (each CTC token spans hop * stack * 2^|subsample_layers| = 1280 samples). Every frame is decoded exactly once, which also removes the word-duplication artifacts the windowed approach showed at small left contexts. left_context_sec is gone (the carry replaces it); center_chunk_sec remains as the partial cadence. End-of-turn on the reference sample: 6 CPU threads 200 -> 146 ms, Vulkan 92 -> 72 ms; 1-3 threads unchanged to slightly better (1007/347/261 ms), all with identical partial cadence. One boundary word at chunk edges can still be dropped (truncated right context — inherent to streaming CTC without lookahead); a lookahead-hold refinement is a possible follow-up.
…ain) matmul_weight_type now also accepts q4_0/q4_1/q5_0/q5_1/q4_k/q5_k/q6_k (the store re-quantizes from the source weights via dequant -> ggml_quantize_chunk). Measured on a Comet Lake i5-10400 (no VNNI): q4_k is 29% SLOWER than q8_0 offline at 1 thread, q4_0 8% slower, and granite streaming is unchanged within noise - the q8_0 GEMM is already the efficient path without int8 dot-product instructions. Accuracy was unchanged on the reference samples even at q4_0. Plumbing kept for VNNI-capable CPUs; documented as a negative result.
The streaming chunk producer only encoded a chunk when the remaining waveform still filled a full 32-mel-frame window; the tail (up to ~330 ms) was silently dropped. Turns whose speech ended inside that tail lost the last word(s) - short utterances like 'Hello' came back empty, while long phrases only lost trailing silence and looked fine. Below one full first window (252 ms) the request even failed outright. Zero-pad the tail to the full window (the reference processor right-pads the final chunk) and encode one last chunk at end of stream; pad the waveform up to the first required chunk instead of throwing. Also trace decoded token ids at TRACE level for empty-transcript diagnosis.
… lookahead-0 crashes - /live gains lookahead_tokens (model right-context; chunk duration = (lookahead+1) x 80 ms on nemotron) and stream_chunk_ms (ingest batching bound) query params. The session already honored lookahead_tokens as a request option; the /live edge just never exposed it, and the ingest layer batched a full policy second per read regardless of the model window. lookahead < 1 is rejected: a single-frame first window decodes garbage (the reference chunking is not validated for it here). - Fix the streaming window builder for windows that precede the signal start (lookahead 0: the second window begins at hop - n_fft/2 = -96): zero-pad the left context like the first chunk's center pad instead of indexing before begin() (heap corruption / ggml abort). - Streaming encoder graphs get their own smaller default arena (256 MB): the prefix ladder builds one graph variant per prefix step and the offline graph's 1 GB default times ~56 variants exhausted commit; the warm-up ladder is also capped at 16 variants, the rest build lazily. - Cache hand-off between graph variants now checks layouts by name and hands attention caches over through the host staging path. - TRACE: decoder token ids and encoder stream stage markers for empty-transcript / mid-stream-crash diagnosis.
…; reject lookahead 0 - Accepted sockets for the live endpoint disable Nagle: small SSE events waited for delayed-ACKs while the client was still uploading, so partials generated during speech reached the client only at end of turn (measured first-partial-on-wire for a 1.9 s phrase: 1861 -> 526 ms at 1 CPU thread). - lookahead_tokens=1 (160 ms chunks) decodes correctly (validated against three phrases at two ingest cadences) although the GGUF's embedded supported list omits it; the model card declares 80-1120 ms chunks as runtime knobs. EOT->final on a 1.9 s phrase: 423 -> 277 ms (1 thread), 130 -> 100 ms (6 threads), 84 -> 73 ms (Vulkan). - lookahead_tokens=0 (80 ms chunks) stays rejected: measured wrong text (empty / wrong-language output) and no speed benefit - the single-frame first window is not covered by the reference chunking validation.
Contributor
Author
Follow-up fixes pushed to this branch (2026-09-18)All measured on the same machine (6C/12T), real-time-paced streaming upload, median of 5 turns.
Ingest cadence below the model window size changes nothing (window completion drives the decode schedule), so the plugin-recommended config stays |
The incremental streaming schedule emits every encoded frame the moment its chunk is done, so the frames at each chunk's end are encoded without their lookahead right-context. The reference avoids this by re-encoding the right context every step (~1.8-4x encoder compute, not viable on CPU). Long turns tolerate the approximation (most frames sit mid-chunk), but short utterances put most of their speech frames at chunk ends: a 0.55 s 'Okay' turn decoded '' on the live path while the offline endpoint decoded the identical audio to 'Okay.' - and a 0.73 s 'Hello' came back 'Hel' on lookahead 1. finalize() now re-decodes turns of up to 1 s (and any turn whose incremental final came back blank) through the offline encoder - one full-context pass over the buffered turn. The cost is bounded by the turn length (measured ~310-370 ms EOT at 2 threads for sub-second turns), long turns keep the streaming fast path, and live partials are unaffected. Note run_streaming_audio (the buffered streaming path) shares the chunked graph and does NOT fix this - the offline encoder is required.
Contributor
Author
Follow-up: short utterances came back empty on the live path — root cause and fix (
|
| turn (preroll 0.2 s + speech + tail) | live (la3) | offline |
|---|---|---|
| 0.55 s "Okay" | '' |
Okay. |
| 0.53 s "Hello" | '' |
Hello |
| 0.47 s "Yes" | '' |
Yes. |
Long turns are unaffected (most frames sit mid-chunk). Reducing the chunk to 160 ms (lookahead_tokens=1) narrows the loss but does not close it ('Hel').
Fix: finalize() re-decodes turns of up to 1 s — and any turn whose incremental final is blank — through the offline encoder (one full-context pass over the buffered turn; note run_streaming_audio does not help, it shares the chunked graph). Cost is bounded by the turn length: 310-370 ms EOT at 2 threads for sub-second turns; long turns keep the streaming fast path (1.9 s phrase: 99-131 ms, 5 live partials). The full 18-case short-turn matrix passes with the fix (the only two failures are zero-preroll turns, which fail offline too).
…v sizing, ladder budget; gate stays closed la0 (80 ms chunks) is the only schedule that is exact AND realtime on one CPU thread (emit-all has no chunk-end truncation when lookahead is 0), so this lays the groundwork: - The first streaming window now covers at least one full encoded frame (subsampling_factor mel frames): at la0 the previous single-mel-frame first window made the subsampling convs compute encoded frame 0 from zero-padded cache input. - The first chunk's mask sizes are derived from the actual conv prepend (zero frame + cache = 2 frames of left context, no right pad). The centered formula assumed a right pad that does not exist and overcounted for even mel counts (mask 5 vs conv 4 at la0's 8-mel window); odd counts (la3's 25-mel window) are unaffected. - prepare_streaming_capacity mirrors the session's first-window size so the first chunk hits the prebuilt graph. - Streaming graph arena defaults to 96 MB and the prefix-ladder warm-up is bounded by a 6 GB commit budget instead of a variant count. - warm bench: the streaming flow now calls start_stream (the missing call left the window geometry zero-initialized); MSVC setenv fix. The /live endpoint still rejects lookahead_tokens < 1: with the geometry fixed, the incremental transducer under-emits at 1-frame chunks (measured 'Hel'/'' finals at 1-2 threads) - the per-chunk emission needs a dedicated investigation before la0 can ship.
…ssion open la0 status after the deep-dive: - The encoder now produces varying, input-tracking outputs at 1-frame chunks (the la0 groundwork geometry + the key-only 1-frame prefix KV trim fixed the constant-garbage outputs). - The offline encoder at la0 decodes all phrases correctly. - The incremental transducer still under-emits at 1-frame chunks (live partials blank; finals rely on the blank-final offline re-decode, which succeeds for turns with >=150 ms of trailing audio — the mic test client carries 250 ms). - Evidence trail for the remaining bug: mel inputs identical to offline, KV cache copies bit-perfect, attention bias allows all keys, yet the encoder outputs degrade with the copied prefix length (corruption grows with prefix frames; a single-key prefix approximates offline closely). Suspect: the attention prefix mechanism (positions/KV reuse) at 1-frame chunks. - /live accepts lookahead_tokens=0; the blank-final offline re-decode covers the incremental gap. The client default remains lookahead 3.
…n bug
Root cause of the la0 under-emission, found by frame-level A/B tracing
(mel inputs, encoder outputs, KV cache contents, incremental vs offline):
The graph allocator aliased the streaming graph's output caches: in the
first chunk's graph, next_attention_key_cache, next_attention_value_cache
and next_subsampling_cache0 shared ONE data pointer (verified: identical
pointers, and the 'KV' read-back returned the subsampling cache's padded
mel content). The cache hand-off copied bit-perfect copies of clobbered
memory, so every chunk after the first encoded from corrupted state —
outputs went flat (~0.19 constant) and the transducer emitted blanks.
Fix: chain a zero-weighted scalar dependency (ggml_sum + ggml_scale(0) +
ggml_add1) from every next-cache tensor into the graph's output node.
The allocator must keep every cache alive until the graph ends; the
buffers stay distinct. The scales are exactly zero, so the encoder
output is mathematically unchanged.
Result at lookahead 0 (80 ms chunks, the exact emit-all schedule):
- The incremental transducer now emits LIVE (deltas stream during
speech; measured tok 'Hel'+'lo' at frame 7).
- End-of-turn -> final latency drops to 89-321 ms on 1-2 CPU threads
(was 600-1200 ms buffered).
- 5/7 phrases fully correct at 1-2 threads; the two remaining failures
('Hel', 'Tell me a sto') lose the final syllable carried by the
zero-padded flush window — the next investigation target.
Also: the decoder per-frame debug traces and the bisect env switches are
removed; the warm bench keeps its start_stream fix.
…t transformers reference - Session + ladder use a 4-encoded-frame chunk floor at lookahead 0 (320ms native chunks) so CPU keeps up with realtime; la3 keeps its existing geometry (max(la+1,4)*8 mel matches 8*(la+1) for la>=3). - Verified against the official reference (transformers nemotron3_5_asr, NVIDIA-contributed): mel windows bit-identical, first-chunk encoder output exact, transcripts match on the short- phrase set; /live flip-flopping is gone (25/25 deterministic). - Opt-in NEMOTRON_DUMP_CHUNKS per-chunk mel/encoder dump for reference diffs. - Remove temporary [INC]/[OFF] decoder traces and unused includes.
Contributor
Author
Lookahead-0 verified against the official reference implementationValidated this branch's nemotron streaming against the NVIDIA-contributed reference in transformers ( Findings:
Lookahead 0 native 320 ms vs lookahead 3 (same binary, end-of-turn -> final transcript, realtime-paced upload):
(All ms, median of 3.) Known edge: a quiet trailing syllable can be dropped at la0 (story -> "Tell me a sto") that la3 still catches; the f32 reference decodes the same audio fully. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ASR: real native streaming for Nemotron 3.5 + Granite Speech (and a granite graph cache)
Summary
Two ASR families shipped a
streamingmode that never streamed. Bothprocess_audio_chunkimplementations only buffered incoming audio and returned an empty event — all decoding
happened in
finalize()over the whole buffer. Clients of/v1/audio/transcriptions/livetherefore saw zero partials while the user was speaking (nemotron emitted its deltas as
one burst after the stream closed; granite emitted nothing at all until the final result),
which defeats the point of the live endpoint.
This PR makes both sessions stream the way the model publishers intend, reusing machinery
that was already in the tree:
nemotron_asr — incremental chunk pipeline
The decoder was already fully incremental inside
decode_streaming()(stateful greedyTDT loop with
on_text_deltaper token), and the encoder already had cache-awareencode_stream_chunk()+NemotronEncoderStreamState. The session simply never used themper chunk. It now:
subsampling × (lookahead + 1)mel frames, the publisher'sgeometry) through
encode_stream_chunk()with a persistentNemotronEncoderStreamState,transcript.text.deltapartials per chunk,finalize()so the tail is encoded instead ofdropped (previously up to
win_length + hop·(lookahead+1)samples of trailing audionever reached the encoder),
(
begin_stream_decode()/decode_stream_chunk()/finish_stream_decode()), withdecode_streaming()reimplemented as a thin wrapper over it — pull-style callers keeptheir exact behavior.
granite5asr — chunked CTC streaming (the publisher's recipe)
Granite5ASRStreamingSessionwas a stub: buffer +transcribe_audio()at close. It nowimplements chunked TurboCTC inference: every center chunk
(
granite5asr.center_chunk_sec, default 1 s) is re-encoded together with its left context(
granite5asr.left_context_sec, default 2 s) and CTC-greedy decoded with a continuouscollapse across window boundaries (running previous-token state), streaming a partial per
chunk. The window→CTC-frame mapping accounts for the full token stride
(
hop_length × stack_factor × 2^|subsample_layers|— the encoder subsamples twice), whichwas the subtle part: mapping frames at the mel rate silently decodes nothing from window 2
onwards. Both options are declared in
model_specs/granite5asr.json.granite5asr CPU optimization: encoder graph cache + cache-aware streaming
transcribe_features()rebuilt the entire 16-block conformer graph (context + gallocr +graph) on every call. Chunked streaming re-decodes the same window shape every chunk and
repeated offline turns cluster around a few lengths, so graphs are now cached per input
shape (small LRU, 6 entries).
More importantly, the streaming session no longer re-encodes a left-context window per
chunk. Granite's attention is block-local (
context_size128-frame blocks attend onlywithin themselves — the rel-pos slice is block-local too), so a chunk needs no attention
cache and no left context: each center chunk is encoded exactly once, prepended only with
a 0.3 s waveform carry covering the two stride-2 subsample blocks' convolution reach, the
avg-pool pair alignment, and the frontend's reflect padding; tokens fully inside the carry
are skipped (one CTC token = hop x stack x 2^|subsample_layers| = 1280 samples). Every
frame is decoded exactly once, which also eliminates the word-duplication artifacts
windowed re-encoding showed at small left contexts.
left_context_secis removed;center_chunk_secstays as the partial cadence (center 2 s trades ~20-25% moreend-of-turn latency for ~30% less total compute).
Measured behavior (v0.8.0 → this branch, same machine)
6.4 s English sample,
/v1/audio/transcriptions/live, paced realtime upload, SSE timingclient-side; "end-of-turn" = last audio byte →
transcript.text.done:Offline transcripts are unchanged (checked on the reference samples), and offline RTFs are
unchanged (the granite graph cache only removes fixed build overhead; on repeated
same-shape requests the encoder no longer rebuilds the graph at all).
CPU profiling at low thread counts (1-3 threads)
Both models are compute-bound with no significant waste left in the graph implementations:
granite5asr.center_chunk_sec/left_context_secknobs trade this off: center 2 s cuts total streaming encoder compute ~30% (3390 vs 4807 ms @ 1T, 4 windows vs 7) at equal accuracy, but grows the final flush window and adds ~20-25% end-of-turn latency, so the 1 s default stays for latency; left contexts below 2 s duplicate words across window boundaries. These options are currently only reachable through the CLI (--session-option); the server does not expose config/request session options — worth addressing upstream.Remaining, larger follow-ups (not in this PR):
Sub-q8_0 encoder kernelsmeasured: no gain on AVX2 without VNNI. Implemented and benchmarked (weight re-quantization to q4_0/q4_k/q6_k via the existing storage pipeline): nemotron offline @ 1 thread q8_0 2623 ms vs q4_k 3376 ms (slower) vs q4_0 2825 ms; granite streaming unchanged within noise; transcripts unchanged even at q4_0. Comet Lake has no int8 dot-product instructions, so q8_0 GEMM is already the efficient path. The weight-type plumbing is kept (opt-in) for VNNI-capable CPUs.audio_chunk_mode=vad; measured neutral on dense speech, helps pause-heavy real turns).Known issue found while testing: short-turn empty text (pre-existing, upstream)
While validating the plugin integration end-to-end we found that the
streaming decode under-contexts each window's trailing
lookaheadframes: a 0.85 s "Hello" decodes only blank tokens over /live (the word
emerges only when a window tail finally provides future context), while
the offline full-context decode of the same audio is correct. This
reproduces on the SHIPPED v0.8.0 build as well (both 0.85 s and 1.3 s
clips finalize empty in streaming mode), so it predates this PR - the
PR's per-chunk streaming makes the mid-stream blanks visible as missing
partials instead of a single empty final.
The proper fix is center-only emission with overlapping windows (each
window = [center | lookahead] encoded frames, advance by center only,
the lookahead region re-encoded by the next window). We prototyped it;
it requires the encoder graph builder to support overlapping window
advances (the current attention/prefix cache bookkeeping assumes
contiguous non-overlapping windows - a time_mask dim mismatch arises
for the new shapes), so it is out of scope here and filed as follow-up
work. Downstream, the integrating application should treat the offline
POST as the authoritative final transcript for each turn and use /live
only for live partials.
Notes
decode_streaming()keeps its old signature and semantics; the new incremental API isadditive.
for streams that end before the first native chunk.
window disagrees; in that case the current full transcript is re-emitted as the delta
(documented in the code). The final transcript is always the authoritative text.
Requesting review — happy to adjust the granite defaults or fold the incremental decoder
API differently if you prefer a different split.