feat(tts): Chatterbox Nano backend — 110M GPT2-small T3 + 2-step meanflow (English, paralinguistic tags) - #916
Merged
Conversation
…CoreML chain Adds ChatterboxNanoManager/-Synthesizer/-Models/-Tokenizer on top of the Multilingual backend, with the Nano host deltas: batch-1 decode (no CFG combine, no alignment analyzer), GPT2 byte-level BPE from the upstream slow-format assets (vocab.json + merges.txt + added_tokens.json, with added-token splitting so the 20 paralinguistic tags map to ids 50257-50275), bare-embedding prefill (GPT2 wpe is applied in-graph), single-BOS context, turbo sampling order (temperature -> top-k 1000 -> top-p 0.95 -> repetition penalty 1.2, applied to filtered logits, first-step BOS penalty matched), and the upstream 3x silence-token append before the meanflow flow call. The strided-fp16 floatBuffer and MLState KV seeding move to a shared ChatterboxMLSupport used by both synthesizers. Tokenizer parity is pinned against transformers 5.2.0 reference ids (incl. space-before-tag -> lone 220, contractions, double spaces). Verified on M5 Pro (release build, models from FluidInference/chatterbox-nano-coreml, built-in voice): 8.08 s audio in 1.35-1.51 s => RTFx 5.4-6.0x wall; Parakeet v3 round-trips the outputs verbatim and [chuckle] renders audibly. Debug builds sit at ~16 ms/token because of the unoptimized per-step sampling sorts - benchmark in release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #907 (diff is Nano-only; retarget to main after #907 merges).
Swift backend for FluidInference/chatterbox-nano-coreml, converted in mobius #91. Requested on Discord: the 500M Multilingual chain is heavy for older iPhones — Nano is the on-device-sized variant (110M T3, distilled 2-step mel decoder), English-only with inline paralinguistic tags (
[laugh],[chuckle],[sigh], …).What's here
ChatterboxNanoManager(public actor, macOS 15+/iOS 18+) +ChatterboxNanoSynthesizer: prefill → MLState stateful decode → meanflow flow → HiFT. Host deltas vs the MTL backend: batch-1 (no CFG combine, no alignment analyzer), bare-embedding prefill (GPT2wpeis in-graph), single BOS, turbo sampling order (temperature → top-k 1000 → top-p 0.95 → repetition penalty after filtering; first-step BOS penalty matched to upstream), 3× silence tokens appended before the flow call.ChatterboxNanoTokenizer: GPT2 byte-level BPE from the upstream slow-format assets (vocab.json+merges.txt+added_tokens.json) with added-token splitting;puncNormis the turbo variant (no"..."/" - "/";"rewrites, ASCII-only enders).ChatterboxMLSupport(strided-fp16floatBuffer+ MLState KV seeding) now used by both synthesizers.Repo.chatterboxNano,ModelNames.ChatterboxNano),TtsBackend.chatterboxNano, CLItts --backend chatterbox-nano+tts-benchmark --backend chatterbox-nano.220+ tag id, contractions, digits, double spaces) — skip when assets aren't cached.Verified (M5 Pro, release build, built-in voice)
tts --backend chatterbox-nano(8.08 s sentence)[chuckle]tag sentenceWeight footprint ~711 MB fp16 on disk (vs ~2.3 GB Multilingual).
Follow-ups