Explore llama.cpp/ggml quantization kernels for our GGUF integration

[moon-bot-app]

Transformers recently landed support for running llama.cpp / ggml quantization schemes directly, backed by a set of ggml kernels distributed through the kernels library. Write-up: https://huggingface.co/blog/transformers-llama-cpp-quants

This looks potentially interesting for our GGUF integration in Diffusers:

• Today our GGUF support dequantizes on the fly; the ggml kernels could let us run quantized ops natively instead, which should help both memory and speed.
• It could also open up nicer MPS / local-inference angles, which is a frequent ask for Diffusers users.

Relevant kernels (currently being moved from transformers-community to the ggml-org org):

• https://huggingface.co/kernels/transformers-community/ggml-quantization
• https://huggingface.co/kernels/transformers-community/ggml-gated-delta-net
• https://huggingface.co/kernels/transformers-community/ggml-norm
• https://huggingface.co/kernels/transformers-community/ggml-attn

Next steps to figure out:

1. Which of these kernels are actually applicable to diffusion model architectures (quantization/norm most likely; attn depends on our attention dispatch).
2. Whether they can be plugged into the existing GGUFQuantizer path via the kernels library.
3. Benchmark quality/speed/memory vs. the current dequantize-on-load behaviour, including on MPS.

---

Requested by Sayak Paul - Slack thread - Agent trace

[iwr-redmond]

While Llama.cpp proper cannot run models with media outputs, there is a longstanding downstream project that supports inference with a variety of diffusion models here. Llama.cpp contributor @ServeurpersoCom also maintains downstream GGML projects for Ace-Step and MiniMax Music.