zipvoice: add k2-fsa ZipVoice community TTS with zh/en frontend

[davidxifeng]

Zero-shot voice cloning on a TTSZipformer flow-matching backbone with a Vocos mel-24kHz vocoder; supports the distilled 8-step variant (guidance-scale embedding) and the base model (batched CFG).

• TTSZipformer ggml graphs: text encoder with duration-ratio conditioning, U-Net flow-matching decoder (relative-position attention, non-linear attention, downsampling stacks) in the physical [C,T,B] convention
• Vocos CPU vocoder (ConvNeXt + iSTFT); single self-contained GGUF packaging: model.* + vocos.* namespaces with the tokens/config and the zh frontend sidecars embedded, so the hosted package is one file (davidxifeng/zipvoice-gguf, huggingface_snapshot download); the spec is GGUF-only and the naming follows the repo convention (-.gguf inside a -GGUF directory)
• Chinese/English text frontend (tokenizer=emilia) reproducing the upstream EmiliaTokenizer: framework Chinese normalization, punctuation mapping, jieba segmentation via vendored cppjieba headers (dictionaries are not vendored: export_zipvoice_zh_dict.py downloads them from the pinned cppjieba commit, SHA-256 verified), pypinyin readings from baked zh_chars/zh_phrases/zh_syllables sidecars, per-word tone sandhi, espeak-ng for English runs; tokenizer=espeak|simple kept as alternatives
• Conversion tools: convert_zipvoice.py (torch -> safetensors -> GGUF, staging the frontend sidecars into the converter root) and the dict export script
• Session loads the spec-resolved resource bundle so embedded sidecars materialize and resolve from the downloaded file alone; direct API callers (parity harnesses) fall back to loose files next to the checkpoint
• Verification: full-stage parity against the reference PyTorch implementation (encoder taps, text conditioning, sampled features, single+batched velocity, fbank, vocoder audio) and token-for-token frontend parity against the upstream tokenizer; end-to-end zh/en/mixed synthesis ASR-verified from the hosted package
• Docs: docs/community_models/zipvoice.md, catalog and README rows

[0xShug0]

@davidxifeng Thank you for contirbuting a new model! Serveral comments:

1. Please remove the new vendored cppjieba library entirely. Audio.cpp currently avoids vendoring language-specific text-processing libraries such as WeText because they add substantial maintenance cost. ZipVoice uses only a small portion of cppjieba, while most of the added headers are unused.

A better approach is to follow Kokoro’s g2p_multilingual.cpp: implement the required Jieba DAG segmentation and four-state HMM locally within the ZipVoice model. Keep the implementation model-local, include only the required logic, preserve the appropriate license attribution, and embed the dictionaries in the GGUF as model resources.

2. Please use the framework’s ggml operation wrappers/helpers wherever equivalent wrappers already exist, instead of calling raw ggml operations directly. Some wrappers provide backend-specific layout and contiguity handling, which improves portability across different backends.

3. Backend selection is ignored in the PR. The implementation always initializes the CPU backend. CUDA, Vulkan, Metal, and other requested backends silently run on CPU.

4. Vocos is on CPU. It should be replaced with a graph by reusing or generalizing the existing F5 TTS Vocos graph and framework ConvNeXt/iSTFT components.

5. Use the framework text chunker for long-form synthesis.

6. Avoid process-global static caches. Make model weights, Vocos weights, and runtime graphs session-owned so model unload actually releases their memory.

7. Please replace the unbounded graph maps with session-owned runtime::CacheSlots.

8. Handle multichannel reference audio through the framework mono conversion and resampling helper.

Please add performance mertics. The target RTF is < 1 for community models.

[davidxifeng]

@0xShug0 Thank you for the detailed review! 😄

I’ve addressed the comments: removed vendored cppjieba
in favor of a model-local DAG/HMM implementation with license attribution and GGUF
dictionary resources; adopted framework operation helpers, backend selection, graph-based
Vocos, text chunking, session-owned weights and bounded runtime::CacheSlots, and
multichannel audio conversion/resampling.

I’ve also added performance metrics. On Apple M3 with Metal, the two tested cases achieved
RTF 0.2457 and 0.1816, both below the RTF < 1 target. I’ll include the results, benchmark
commands, and parameters below.

Tested ZipVoice-Distill on Apple M3 / Metal, Release build, 8 CPU threads
(2026-09-18). Both cases meet the community-model target of RTF < 1.

Reference	Generated audio	Mean synthesis time	RTF
Male	19.115 s	4.697 s	0.2457
Female	12.843 s	2.332 s	0.1816

Parameters: zipvoice-distill-orig.gguf with bundled Vocos, 8 Euler steps,
guidance scale 3, time shift 0.5, speed 1, seed 666, feature scale
0.1, target RMS 0.1, Emilia tokenizer, and 24 kHz output.

Please take another look when you have a chance. Thanks!

[davidxifeng]

Next, I'd like to work on:

• Optimizing performance on the ggml Metal backend. I only have a Mac development machine, so I don't have access to a CUDA environment.
• Quantizing the model and evaluating the results, including inference speed, accuracy degradation, and other trade-offs.
• Porting the newly released Confucius4-R2T2 model. It looks particularly well suited for speech-input / voice-typing use cases, so I'd like to experiment with running it efficiently on macOS as well.

[0xShug0]

@davidxifeng Thanks for the update! The code is in much better shape now.

Could you exclude the ggml changes from this PR? ggml changes need to be submitted in separate PRs and should be additive in nature (if not should provide regression tests for affected models). I think the current performance is already good enough (my RTF < 1 is for CUDA :) ). Usually Metal is much slower than CUDA.

Can’t test it right now, but I’ll do so as soon as I can.

[0xShug0]

@davidxifeng The broader model changes are substantially improved. Only three minor issues to fix before merge:

(1) Remove ggml changes.

(2) The spec defines only zipvoice_distill_gguf pointing to the F32 file. python3 tools/model_manager_v2.py install zipvoice_distill_q8_0 will produce

error: unknown package or family: zipvoice_distill_q8_0

(3) ZipVoice loads every model tensor through load_f32_tensor(), and Vocos does the same. Therefore Q8 weights are expanded to F32 during loading.

Please keep the changes scoped to the three comments so I can just review the delta. The other changes can be follow-up PRs.

[davidxifeng]

@0xShug0 Thanks for your reviewing!

Updated the PR to address all three comments:

• removed the ggml changes
• registered zipvoice_distill_q8_0
• preserved native quantized weights in ZipVoice and Vocos.

F32 parity and Q8 CPU/Metal synthesis checks passed.
Please take another look when you have time. Thanks!