Conversation
CPY, SET, DUP and CONT accepted f32 conversions and same-type copies on Metal, but not the two cross combinations of bf16 and f16, so a graph copying between them could not be scheduled on the backend. Instantiate kernel_cpy_f16_bf16 and kernel_cpy_bf16_f16 (contiguous and strided) under GGML_METAL_HAS_BF16, and accept the F16<->BF16 pairs in ggml_metal_device_supports_op.
BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the reference implementation. Enable the same policy on Metal and use a bf16 KV cache there as well. Unlike CUDA/HIP/Vulkan, Metal has no fused round-to-bf16 unary op, so fused_round stays disabled and the cast is a separate graph node.
|
@gqf2008 In my test the policy change in breeze tts caused 22% to 25% regression in the AR compoent (mac mini M4). Could you do a performacne regression check? |
|
Confirmed — the regression reproduces on Metal here, at about the same magnitude you measured. Setup
Result (median, 10 runs per arm)
Per-iteration AR ratio (head/base): 1.72, 0.95, 1.40, 1.22, 1.07, 1.20, 1.72, 1.42, 1.58, 0.76 → median 1.31 (1.22 excluding the cold run 1). AR ms per run, main: 21493, 24378, 19772, 19300, 21434, 17569, 20381, 21661, 22294, 33026; this PR: 36945, 23101, 27734, 23612, 22922, 21154, 35062, 30854, 35324, 25160. MechanismThe cost is concentrated in the backbone (prefill +29%, decode +22%) — exactly the part that picks up the per-cell bf16 activation casts and the bf16 KV cache. The Mimi speech decoder is untouched (+4%, inside the noise). Metal has no fused round-to-bf16 unary op: Options
Happy to do (2) or (3); tell me which you prefer. |
Metal was the only GPU backend without GGML_UNARY_OP_ROUND_BF16, so Breeze's bf16 activation policy fell back to an f32 -> bf16 -> f32 cast pair at every rounding point (~14 per layer per decode step). That is the regression the maintainer measured on a Mac mini M4 (+22-25% on the AR component, reproduced here at +20% per generated frame with paired interleaved runs). Add the fused op for Metal: - kernel_unary_impl gains OP_UNARY_NUM_ROUND_BF16. The rounding mirrors ggml_compute_fp32_to_bf16 (round-to-nearest-even at bit 16, NaNs forced quiet) using integer math, so it does not depend on the device exposing the bfloat type; float4 shares the same helper. - New kernel_unary_f16_f32(_4) / kernel_unary_bf16_f32(_4) instantiations: the op accepts f32/f16/bf16 sources and always produces f32. Reading a bf16 source still requires hardware bf16 support; f32/f16 sources work anywhere. - breeze_tts's activation policy now enables fused_round on Metal as well. Verified on macOS 26.5.2 (M4, Metal), BreezeTTS 2 Q8_0, fixed text and seed: the generated audio is byte-identical to the unfused policy arm (3/3 runs), so the fused op is numerically the same rounding. Paired interleaved runs against the unfused build give a median AR ratio of 0.86 over 14 pairs (0.80 over the first 7, before the machine picked up background load) - i.e. the fused op takes back most of the ~20% the cast pairs cost, matching what the same op was worth on CUDA (19-29% of GPU time in 0xShug0#393).
|
Follow-up: I implemented option 2 from my earlier analysis — What it adds
Verification (macOS 26.5.2, M4, Metal, BreezeTTS 2 Q8_0)
What is leftWith the fused op in place, the remaining cost of the policy on Metal is the bf16 KV cache plus the extra graph structure, not the casts: a per-generated-frame comparison against the policy-off arm still shows roughly +20%. That is the same knob CUDA/Vulkan deliberately keep at f16 on non-sm80 parts, so if you want I can look at whether Metal should keep an f16 KV cache and only use the bf16 activation rounding (that would isolate the two effects). Also happy to split the fused op into its own PR if you would rather land the backend feature independently of the breeze policy change. |
|
One more data point while the measurement rig was set up: I tried keeping the f16 KV cache on Metal (the way CUDA and Vulkan do) while keeping the bf16 activation rounding, to see whether the KV cache type explains the residual gap against the policy-off arm. Result: no measurable difference on this machine. Same text/seed, same output length (51 frames), 3 paired interleaved runs:
Paired ratios (f16/bf16) were 0.93 / 1.29 / 1.20 — i.e. inside the noise floor (this fanless M4 Air swings ±50% run to run). So I did not include that change: the data does not support it, and the bf16 KV cache is what matches the reference implementation anyway. One thing the logs do show, which may be useful if you want to chase the rest: the AR loop is dominated by the depth decoder and sampling, not the backbone — backbone decode is ~32 ms of the ~250 ms per generated frame, the remainder is depth decoding/sampling. That is also why the fused op pays off as much as it does: the cast pairs were per-layer/per-step on both stacks. I'll leave it here rather than push a change the numbers don't support; if you want the residual gap chased properly, it needs a quieter machine (or a much longer interleaved run) — happy to do that on request. |
|
@gqf2008 On the other backends, the additional optimizations amortize the BF16 cast cost to some extent, while on Mac the regression is more visible. I can confirm the 20% regression on my Mac Mini. For now, I’d prefer to add an option and update the docs so Mac users can explicitly opt in until the performance gap is fully addressed. |
2 participants
Summary
BreezeTTS rounds activations to bf16 on CUDA/HIP/Vulkan to match the reference implementation. Metal was excluded from that policy; this enables it there and uses a bf16 KV cache as well.
The branch also adds the two missing bf16<->f16 copy kernels to the Metal backend, so a graph that copies between those types can be scheduled on Metal instead of falling back.
Changes
ggml-metal: add bf16<->f16 copy kernelsCPY,SET,DUPandCONTaccepted f32 conversions and same-type copies on Metal, but not the two cross combinations of bf16 and f16.kernel_cpy_f16_bf16andkernel_cpy_bf16_f16(contiguous and strided) underGGML_METAL_HAS_BF16, and accept theF16<->BF16pairs inggml_metal_device_supports_op.breeze: enable bf16 activation rounding on Metalfused_roundstays disabled and the cast is a separate graph node.Verification
On macOS (Apple M4, Metal 4) with both targets built (
audiocpp_cli,audiocpp_server):audiocpp_cli --task tts --family breeze_tts --model breeze-tts-2-q8_0.gguf --backend metalruns to completion and produces audio.kernel_cpy_bf16_bf16,kernel_cpy_bf16_f32,kernel_cpy_f32_bf16andkernel_set_rows_bf16_i32.ggml-vulkan: add bf16<->f32/f16 cpy pipelines), not as a requirement of the BreezeTTS change: enabling the policy alone builds and runs identically.