feat(tts): Chatterbox backends — Multilingual + Nano (T3 + S3Gen CoreML chains) (#49) - #907
Merged
Merged
Conversation
…49) New beta TTS backend for ResembleAI Chatterbox Multilingual (FluidInference/chatterbox-multilingual-coreml, converted in mobius models/tts/chatterbox/coreml). macOS 15+/iOS 18+ (MLState KV decode). Pipeline: grapheme-BPE tokenizer (18 of 23 upstream languages; zh/ja/he/ ko/ru need unported text transforms) → host embedding prep from tables.safetensors (CFG text-zero before positional add, double-BOS context, both upstream-faithful) → T3-Prefill → stateful AR decode (CFG combine, AlignmentStreamAnalyzer port driving EOS from exported attention-head rows, repetition penalty + min-p + seeded multinomial) → S3Gen flow (500-token bucket, host-seeded CFM noise) → HiFT vocoder (host-seeded SineGen randomness) → 24 kHz. The MLMultiArray marshalling honors output strides and fp16 storage: GPU-backed CoreML outputs arrive as fp16 IOSurfaces with padded row strides (mel [1,80,1000] ships strides [80640,1008,1]) even when the model declares fp32 outputs — a naive contiguous read smears every row, which produced pure noise until floatBuffer/seedState became stride-aware. CLI: fluidaudiocli tts --backend chatterbox --lang de --seed N, plus a tts-benchmark driver. Verified via fluidaudiocli transcribe round-trip: en/de/fr all exact (de cleaner than the upstream PyTorch baseline run). Decode 21-23 ms/token stateful on M5 Pro GPU; flow ~4.7 s is the bottleneck. Never load the T3 packages .cpuOnly (hard-crash, see model card).
PocketTTS Smoke Test ✅
Runtime: 0m6s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Supertonic3 Smoke Test ✅
Runtime: 0m24s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 20s • 2026-09-13T23:47:26.060Z |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m57s • 09/13/2026, 07:45 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 47.1s diarization time • Test runtime: 2m 34s • 09/13/2026, 07:47 PM EST |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 104.5s processing • Test runtime: 1m 57s • 09/13/2026, 08:09 PM EST |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 7m53s • 09/13/2026, 08:01 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
…ouble spaces punc_norm collapses whitespace before the punctuation replacements, so ', ' insertions produce double spaces; verified against the Python reference and consistent with the [7,2,2] double-SPACE ids already in the parity vectors. The Swift port was faithful; the assertion wasn't.
…CoreML chain Adds ChatterboxNanoManager/-Synthesizer/-Models/-Tokenizer on top of the Multilingual backend, with the Nano host deltas: batch-1 decode (no CFG combine, no alignment analyzer), GPT2 byte-level BPE from the upstream slow-format assets (vocab.json + merges.txt + added_tokens.json, with added-token splitting so the 20 paralinguistic tags map to ids 50257-50275), bare-embedding prefill (GPT2 wpe is applied in-graph), single-BOS context, turbo sampling order (temperature -> top-k 1000 -> top-p 0.95 -> repetition penalty 1.2, applied to filtered logits, first-step BOS penalty matched), and the upstream 3x silence-token append before the meanflow flow call. The strided-fp16 floatBuffer and MLState KV seeding move to a shared ChatterboxMLSupport used by both synthesizers. Tokenizer parity is pinned against transformers 5.2.0 reference ids (incl. space-before-tag -> lone 220, contractions, double spaces). Verified on M5 Pro (release build, models from FluidInference/chatterbox-nano-coreml, built-in voice): 8.08 s audio in 1.35-1.51 s => RTFx 5.4-6.0x wall; Parakeet v3 round-trips the outputs verbatim and [chuckle] renders audibly. Debug builds sit at ~16 ms/token because of the unoptimized per-step sampling sorts - benchmark in release.
Alex-Wengg
changed the title
Documentation/TTS/Chatterbox.md covers Multilingual + Nano behind a beta banner (variant matrix, usage, upstream-faithful sampling defaults, the 10 s flow-bucket cap, cpuOnly crash warning, and the debug-build benchmark caveat). Benchmarks.md gains beta-flagged corpus rows, README's TTS bullet links the new doc, and the Repo cases + Nano constants carry the beta note alongside the existing manager/model/TtsBackend ones.
…ing, whitespace norm, honest benchmark presets 1. MTL tokenizer iterated Swift Characters after NFKD, so decomposed pairs (u + U+0308) formed one grapheme cluster and fell to [UNK] — every accented character in de/es/fr/el/hi was damaged. Iterate unicodeScalars; verified the scalar algorithm reproduces the shipped vocab's es/de parity vectors with zero UNKs. Non-skipping regression uses a fixture vocab with no precomposed entries so cluster iteration cannot pass. 2. SafetensorsFile: checked header-length conversion, offset ordering and payload bounds, and dtype byte alignment now throw malformedAsset instead of trapping. Both Models loaders drop + re-fetch aux files once on parse failure (cache checks are existence-only, so a corrupt file previously failed every launch). 3. puncNorm split only on ASCII spaces; upstream Python str.split() collapses all whitespace. Newlines/tabs now normalize identically in both variants (a surviving newline reached the Nano BPE as a token). 4. Chatterbox benchmark drivers pinned every model to .cpuAndGPU but recorded whatever --compute-units was requested. Unsupported presets now warn and logs/JSON report the applied cpuAndGpu config.
…etensors shapes 1. Aux drop-and-refetch now goes through ChatterboxMLSupport.loadAuxWithRecovery, which mirrors ModelHub.loadWithRecovery's contract: offline mode rethrows the parse error BEFORE any deletion (cached files preserved, no network), and cancellation is not treated as corruption. Regression tests cover the offline no-purge/no-refetch path and the online delete-then-retry-once path. 2. SafetensorsFile.table computes shape products with multipliedReportingOverflow — a valid JSON header with shape [Int.max, 2] previously passed the non-negative checks and fatally overflowed at rows*cols, bypassing the recovery catch. Both loaders additionally validate expected tensor dimensions (hidden-size columns, vocab-size rows, 80-mel prompt feat, 192-d x-vector) before any raw-pointer row copies. Tests cover shape overflow, reversed offsets, out-of-payload offsets, huge header length, and misaligned tensor bytes; both real bundles re-verified loading clean through the guards.
Dimension checks move to testable ChatterboxTables.validate statics (single source for both loaders) and now bound every unchecked pointer destination: promptFeat.rows must equal 2x promptTokens.count AND fit the melFrameBucket (a structurally valid [1001, 80] prompt mel previously passed loading and wrote 80 floats past the fixed [1, 1000, 80] flow buffer), and the multilingual positional tables must cover every reachable position (textPos >= prefillLength rows, speechPos >= maxContext rows — a one-row text_pos_emb previously passed and trapped at row(1) on ordinary text). Adversarial regressions cover oversized/mismatched prompt features and undersized positional tables; both real bundles re-verified through the completed guards.
Both tokenizers now expose validate(embeddingRows:), called from loadAux after the tables load: every vocab and added-token id (plus the MTL [UNK] fallback) must sit in 0..<textEmb.rows, since each id later indexes the table with an unchecked row slice — a cached tokenizer with "a": 999999 or an added token id of -1 previously passed loading, bypassed cache recovery, and trapped at synthesis. Non-integer ids in vocab/added-token JSON now throw malformedAsset instead of being silently skipped (which would have re-routed those tokens to [UNK] / dropped them). Regressions cover oversized, negative, in-range, and non-integer ids in both tokenizer formats; real bundles re-verified through the new checks.
…-spaces after ellipsis replacement Whitespace collapses before the punc replacements, so '… ' becomes ', ' (verified against tts_turbo.punc_norm: 'Wait… what: no' -> 'Wait, what, no.'). Same class of expectation fix as 6858d7d on the MTL side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New beta TTS backend for ResembleAI Chatterbox Multilingual via the CoreML conversion published at FluidInference/chatterbox-multilingual-coreml (mobius PR #89). Closes the multilingual half of #49's model requests.
swift run fluidaudiocli tts --backend chatterbox --lang de "Der schnelle braune Fuchs springt über den faulen Hund." --seed 42 swift run fluidaudiocli tts-benchmark --backend chatterbox --corpus minimax-englishRequires macOS 15 / iOS 18 (T3 decode keeps its KV cache in
MLState). Built-in voice only; 18 of the 23 upstream languages (zh/ja/he/ko/ru need language-specific text transforms not yet ported).Pipeline (all host logic mirrors the mobius reference driver)
grapheme-BPE tokenizer (2454 vocab / 265 merges,
[lang]+[SPACE]added tokens, NFKD, upstreampunc_norm) → embedding prep fromtables.safetensors(CFG zeroes the text embedding before the positional add; the prefill context ends with two BOS embeds — both upstream-faithful quirks) →T3-Prefill→ stateful AR decode (CFG combine,AlignmentStreamAnalyzerport consuming the exported attention-head rows for EOS control, repetition penalty 2.0 + min-p 0.05 + seeded multinomial) →Flow-N500(host-seeded CFM noise) →HiFT-T1000(host-seeded SineGen phase/noise) → 24 kHz.Verified via fluidaudiocli (M5 Pro)
transcribe(Parakeet v3) round-trip ontts --backend chatterboxoutput:Benchmark — minimax-english, 100 phrases (M5 Pro, 24 GB)
swift run fluidaudiocli tts-benchmark --backend chatterbox --corpus minimax-englishPer-phrase timings settle to ~real-time once warm; the AR decode (25 tokens/s of audio at ~48 tok/s decode) bounds RTFx near 1×. Follow-up decode speed (ANE per-layer KV, int4) is the lever for >1×.
The debugging story worth knowing
First audio out of the Swift chain was pure noise while every input tensor was bit-identical to the working Python driver. Root cause: GPU-backed CoreML outputs arrive as fp16 IOSurfaces with padded row strides (the [1, 80, 1000] mel ships strides [80640, 1008, 1]) even when the model declares fp32 outputs. Naive
dataPointerreads smear every row.floatBufferis now stride- and dtype-aware and everything (logits, alignment rows, KV seeding, mel, audio) marshals through it. If other backends read GPU outputs via rawdataPointer, they're vulnerable to the same class of bug.Also inherited from the conversion: never load the T3 packages with
.cpuOnly(hard-crash; documented on the model card + inChatterboxModels).Tests
ChatterboxTokenizerTests— punc_norm cases, BPE mechanics on a synthetic vocab, and 5 real-vocab parity vectors generated from the upstream Python tokenizer (skip when the asset isn't cached).ChatterboxAlignmentAnalyzerTests— EOS suppression mid-text, forced EOS on token repetition.Follow-ups
Voice cloning (reference encoders not converted), zh/ja/he/ko/ru text frontends, streaming/chunked synthesis, T3 weight sharing between prefill/decode (2×977 MB today), flow bucket variants for shorter latency.
🤖 Generated with Claude Code
Also in this PR: Chatterbox Nano backend
Second backend added on the same branch (per review preference — one PR): ResembleAI/chatterbox-nano via FluidInference/chatterbox-nano-coreml (mobius PR #91). 110M GPT2-small T3 + distilled 2-step meanflow mel decoder — the on-device-sized variant. English-only, with inline paralinguistic tags (
[laugh],[chuckle],[sigh], …).swift run -c release fluidaudiocli tts --backend chatterbox-nano "Well that went better than expected [chuckle], see you tomorrow." --seed 42 swift run -c release fluidaudiocli tts-benchmark --backend chatterbox-nano --corpus minimax-englishHost deltas vs the Multilingual pipeline: batch-1 decode (no CFG combine, no alignment analyzer), GPT2 byte-level BPE from the upstream slow-format assets (added-token splitting → tag ids 50257–50275; parity pinned against transformers 5.2.0 reference ids in tests), bare-embedding prefill (GPT2
wpeapplied in-graph), single BOS, turbo sampling order (temperature → top-k 1000 → top-p 0.95 → repetition penalty after filtering, first-step BOS penalty matched), 3× silence tokens appended before the flow call. The strided-fp16floatBuffer+ MLState KV seeding moved to a sharedChatterboxMLSupportused by both synthesizers.Verified (M5 Pro, release build, built-in voice)
[chuckle]sentence (tag renders audibly)Note: debug builds report ~16 ms/token — that's the unoptimized per-step sampling sorts, not the model (same
.mlmodelcsteps at ~4 ms). Benchmark with-c release.Follow-ups: voice cloning encoders (upstream conversions), Turbo 350M (config-only in the same wrappers), iOS smoke test. Supersedes #916.
Review fixes (
d5bab600): scalar-based BPE (decomposed accents no longer fall to[UNK]— fixture-vocab regression test cannot pass with grapheme clustering; scalar algorithm re-verified against the shipped vocab's es/de parity vectors, and the German CLI render re-confirmed verbatim post-fix), safetensors header/offset/alignment validation throws instead of trapping + corrupt aux files re-fetch once,puncNormcollapses all whitespace like Pythonstr.split(), and the benchmark drivers now warn on unsupported--compute-unitsand record the appliedcpuAndGpuconfig.