Feature Tracker and FAQ

[0xShug0]

Use this thread to:

Request new features
Pick up open features TODOs
Ask or find answers to common questions

PRs for these features are greatly appreciated! If you want to work on a TODO, please comment first to avoid duplicated effort.

Features

• Support mp3 and other inputs/output
• HTTPS support
• VibeVoice Streaming ASR 1.5B (extending 7B config to support 1.5B)
• Extend MuScriptor to other variants
• YuE2 AR/NAR LoRA support in cli/server
• YuE2 NAR LoRA support in UI
• Check after release 0.8.2: streaming: publish partial text as increments, through one shared publisher #552
• Server: cli flag or env variables to set the default model root.
• NixOS Cachex support (cachix support #580)
• Name based backend selection
• Other server metrics
• Multi-GPU support for pooling VRAM Multi-GPU support for pooling VRAM (e.g. 2× GTX 1070) like llama.cpp? #505. May need a careful model-by-model rollout.
• Better support for Intel GPU and Adreno GPU (Vulkan: maxComputeWorkGroupCount error (Intel GPU & Adreno GPU) #489)

UI:

• Kokoro missing speed ctr
• Adding a History of music generations (currently any new generation removes the previous one in the right panel.)
• Possibility to attach metadata (prompt, lyrics etc) to the audio file, or
• Exporting the metadata as *.json file, full for reproducibility.
• In HeartMuLa having a default 'pop' in tags would serve better than throwing error: tags needed.
• In ACE-step having a default for prompt 'upbeat ...' would also serve better than throwing error: prompt needed.
• Any new run stops the play, it would be better if it leaves play alone and only stops it if the new generation done and the new song ready.
• The newly added -1 = auto label could move next to Duration seconds similar to existing Seed -1 = random.
• Auto-save gens on computer, instead of clicking save.

FAQ

Docker can't run

This is usually a CUDA driver compatibility issue. See https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-toolkit-and-corresponding-driver-versions

Vulkan

• Useful build tips (from @Dev0Lab1) : Tips for Vulkan Build #475
• Use shared VRAM on Windows (from @Dev0Lab1): set "GGML_VK_ALLOW_SYSMEM_FALLBACK=1" HeartMuLa 3B: failed to allocate HeartCodec scalar decoder graph -- Can't use Windows shared VRAM. Workaround: set "GGML_VK_ALLOW_SYSMEM_FALLBACK=1" #434

Nvidia Jetson Orin

• Check this thread: Nvidia Jetson Orin related questions #12

Multi-component models: YuE2, Minimax Music3, AuK

• If you’re using binaries built from source, you may encounter a model directory contains 2 GGUF files error, causing the server/CLI to refuse to load the model. There are two ways to address this:
  (1) Use --model-spec-override /absolute/path/to/model_spec.json
  (2) Build with AUDIOCPP_DEPLOYMENT_BUILD=ON

AuK/Auk Flash

Honest take: only certain tasks this model supports work as expected.

We validate parity with upstream Python for the tested configuration, not whether AuK's output meets every quality expectation. If a result is disappointing, listen to the corresponding Python reference WAV linked below first.

https://huggingface.co/audio-cpp/AuK-Base-and-Flash-GGUF

A similar Python result points to the upstream model's behavior, not necessarily an audio.cpp conversion issue.

#612 also provides human evaluation of the results.

Placeholder...

[blox11]

https://github.com/woongzip1/UniverSR
https://github.com/jusperlee/apollo

Are these two that you would consider implementing? They are more like audio restoration tools.
Also, I appreciate your work on audio.cpp. Juggling a bunch of different music gen/TTS models between their own implementation or comfy-ui was annoying and difficult to get working, but this has been a very easy experience.

I have some other recommendations too (no MuScriptor Large?), but thought it might be a bit rude to just dump a wall of them without any context 😂

[gitchat1]

Is there any chance we could get a whisper check like it is implemented in this project. https://github.com/zeropointnine/tts-audiobook-tool That would improve quality significently and since audio.cpp runs on both AMD as well as Nvidia it would make the comunity less dependent on Nvidia. It would also be pretty important to be able to offload the whisper model to the cpu for those of us with lower end hardware.

[0xShug0]

@blox11 Thanks for the pointer! I’ve added them to the model candidates list. For variants like MuScriptor Large, they’re usually lower priority for me, so I’m hoping the community can pick them up. Most of these extensions should be pretty straightforward since the architecture stays the same and only the model parameters change. My focus is still on extending support for unique model families.

@gitchat1 I don’t quite understand how this feature works. Could you elaborate a bit?

[gitchat1]

I'm not a programmer so you would have to look at the code to know how it's really done. What the feature does in practice is it retranscribes generated segments of audio and checks the transcript against the original text. If the segment passes the check it is processed further if it does not the segment is discarded and regenerated. The programme runs through this cycle up to five times if it can't resolve the error it accepts a flawed segment. Of course the Whisper model in question does also need VRAM when run together with the model on the graphics card, therefore it would be great if you could reallocate that model in this case whisper-turbo to the CPU for memory constraint environments. What would make it even better is if after the programme fails five times it didn't just pick the latest generation but the best of them. This programme has such a feature. https://github.com/petermg/Chatterbox-TTS-Extended I do not know how difficult something like that would be to implement though. Oh yes and in the case of audiobooktool there are five possible acuracy levels the segments need to pass in order to be processed further. The higher the acuracy level the closer the segment has to be to the real text in order to pass.

[0xShug0]

What the feature does in practice is it retranscribes generated segments of audio and checks the transcript against the original text. If the segment passes the check it is processed further if it does not the segment is discarded and regenerated.

I see. This kind of feature should live in applications built on top of audio.cpp. The core engine won’t implement features like this.

[blox11]

Two more that would potentially be quite useful, especially for using the new YuE2 model with stem separation/lyrics transcription respectively.

https://github.com/facebookresearch/sam-audio
https://github.com/tencent-ailab/songprep

[The1Bill]

I would appreciate the ability to not just cache a custom voice in Breeze, but to save one. I've been using breeze-tts-2.cpp (I have no affiliation with it, I just like this feature) and it allows for cloned voices to be saved a .breeze files to obviate re-running the clone job once per session (if it's being cached) or once per call (if it isn't).

Details on Breeze's 'saving' of voices here.

Is this something that can be added either to the CLI or the UI?

[0xShug0]

@The1Bill The server can config voice preset or voice lib https://github.com/0xShug0/audio.cpp/tree/main/app/server#voice-library-voice_dir. Is this what you had in mind?

[The1Bill]

Kind of. That would let me have the voice preset ready to go but it would still have to re-clone the voice for each run, wouldn't it?

Breeze has a neat way of saving these voices as a .breeze file and running an instance with a .breeze file had lower latency than sending a .wav and a transcript when I tested it. The encoded .breeze file that I use for my assistant is ~5kb while the .wav is based on us ~300kb.

If it's not in scope it's no big deal, it isn't like the latency cost of cloning a voice is all that high.

[0xShug0]

@The1Bill No. TTS models have internal cache for processed reference voice to skip reprocessing. E.g., Breeze has a session option reference_cache_slots default to 1. You can increase the number during server lanuch to allow cache multiple reference voices. In additional to that, for TTS models that need repeated voice-clone context, set a model-level default_voice_preset so OpenAI-compatible clients can omit voice_ref and reference_text on each request:

{
  "id": "omnivoice",
  "family": "omnivoice",
  "path": "/absolute/path/to/models/OmniVoice",
  "task": "tts",
  "mode": "offline",
  "default_voice_preset": {
    "voice_ref": "/absolute/path/to/reference.wav",
    "reference_text": "Reference transcript for the reference audio."
  }
}

[coppe4]

Would it be possible to bundle rnnoise and integrate denoising + silence removal to the server tts ui?

Here is the official OmniVoice hf demo with automatic denoising and postprocessing which makes it easier to get good results.

Currently, it seems with audio.cpp i'd need to first denoise the audio and then feed it as input.

[0xShug0]

Would it be possible to bundle rnnoise and integrate denoising + silence removal to the server tts ui?

Here is the official OmniVoice hf demo with automatic denoising and postprocessing which makes it easier to get good results.

Currently, it seems with audio.cpp i'd need to first denoise the audio and then feed it as input.

Let me think...