Add Ideogram4LoraLoaderMixin (LoRA loading for Ideogram4) by linoytsaban · Pull Request #13921 · huggingface/diffusers

linoytsaban · 2026-06-11T12:56:48Z

Adds Ideogram4LoraLoaderMixin and wires it into Ideogram4Pipeline, so Ideogram 4 pipelines can load_lora_weights / save_lora_weights / fuse like the other models.

This is the loading foundation split out of the Ideogram4 work so it can be reviewed/merged on its own. Two follow-ups are stacked on top of this branch:

Ideogram4 lora training #13861 — the DreamBooth LoRA training script (rebased onto this).
Support loading non-diffusers Ideogram4 LoRA checkpoints #13919 — loading non-diffusers (ai-toolkit/ostris, fal) Ideogram4 LoRAs (rebased onto this).

Both depend on this mixin; this PR is independent of the training script's readiness.

HuggingFaceDocBuilderDev · 2026-06-11T13:06:58Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul

Thanks! I think the tests are missing.

support loading non-diffusers Ideogram4 LoRAs

github-actions · 2026-06-13T08:34:45Z

Hi @linoytsaban, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

sayakpaul · 2026-06-15T03:04:37Z

+    @unittest.skip("Ideogram4 does not support call-time LoRA scaling via attention_kwargs.")
+    def test_set_adapters_match_attention_kwargs(self):
+        pass


Why is this case?

sayakpaul

Just one comment on why the pipeline doesn't support attention_kwargs? How are the users supposed to control the LoRA strength? There also seems to be CI failures.

- pipeline: run the text encoder on its parameters' current device, then move features to the execution device, so encode_prompt works under enable_model_cpu_offload. The pipeline calls the text encoder's submodules directly to tap intermediate layers, which bypasses accelerate's onload hook, so the weights stay on CPU while inputs are on the execution device. Fixes test_lora_loading_model_cpu_offload. - tests: override test_lora_fuse_nan to corrupt a weight under Ideogram4's `layers` tower (the base test probes transformer_blocks/blocks/etc.). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

linoytsaban · 2026-06-16T17:09:37Z

@sayakpaul pushed fixes for the two CI failures you flagged — would appreciate your opinion on the pipeline change in particular.

test_lora_loading_model_cpu_offload needed a small pipeline change, not just a test tweak. Ideogram4Pipeline.encode_prompt builds the text conditioning by calling the Qwen3-VL encoder's submodules directly (embed_tokens, each decoder layer, rotary_emb) in order to tap several intermediate layers' hidden states and concatenate them into llm_features. Since it never calls text_encoder(...) (the module's forward), accelerate's enable_model_cpu_offload onload hook — which is attached to that forward — never fires, so the encoder's weights stay on CPU while the inputs are on the execution device → Expected all tensors to be on the same device in the token embedding.

Fix: run the text encoder on the device its parameters currently live on, then move the features to the pipeline's execution device. This is a no-op on the normal (non-offload) path (te_device == device) and only changes the offloaded case. Open to a different approach if you'd prefer (e.g. manually triggering the onload hook), but this felt the least invasive.

test_lora_fuse_nan was test-only: the base test finds a weight to corrupt by probing transformer_blocks/blocks/joint_transformer_blocks/single_transformer_blocks, but Ideogram4's transformer tower is named layers. Overrode it to use layers[0].attention.to_q, matching the Flux.2 / Lumina2 / Z-Image LoRA tests.

sayakpaul · 2026-06-17T01:59:16Z

+        # Run the encoder on the device its parameters currently live on, then move the features to the
+        # pipeline device. encode_prompt calls the text encoder's submodules directly, so under
+        # enable_model_cpu_offload the onload hook never fires and the weights stay on CPU; honoring their
+        # actual device avoids a device mismatch on the token embedding.
+        te_device = self.text_encoder.device
+        token_ids = token_ids.to(te_device)
+        attention_mask = attention_mask.to(te_device)
+        text_position_ids = text_position_ids.to(te_device)


Doing it this way, the te_device would be the accelerator?

sayakpaul · 2026-06-17T01:59:37Z

        )
        text_features = torch.stack(selected, dim=0).permute(1, 2, 3, 0).reshape(batch_size, max_sequence_length, -1)
        text_features = (text_features * attention_mask.to(text_features.dtype).unsqueeze(-1)).to(torch.float32)
+        text_features = text_features.to(device)


Why cast it back?

sayakpaul

Just a single comment.

add Ideogram4LoraLoaderMixin

ecd7d0a

github-actions Bot added documentation Improvements or additions to documentation lora pipelines loaders size/L PR with diff > 200 LOC labels Jun 11, 2026

linoytsaban requested a review from sayakpaul June 11, 2026 13:51

sayakpaul reviewed Jun 12, 2026

View reviewed changes

Support loading non-diffusers Ideogram4 LoRA checkpoints (#13919)

61eaf8a

support loading non-diffusers Ideogram4 LoRAs

joangava approved these changes Jun 13, 2026

View reviewed changes

add Ideogram4 LoRA loader tests

c19c874

github-actions Bot added the tests label Jun 13, 2026

linoytsaban requested a review from sayakpaul June 14, 2026 20:43

Merge remote-tracking branch 'origin/main' into HEAD

e67cd35

sayakpaul reviewed Jun 15, 2026

View reviewed changes

sayakpaul mentioned this pull request Jun 15, 2026

ci: don't remind on prs from admins, etc. #13965

Open

linoytsaban added 2 commits June 15, 2026 12:23

support call-time LoRA scaling via attention_kwargs in Ideogram4

7c9d727

fix and un-skip Ideogram4 LoRA loader tests

f963a63

github-actions Bot added the models label Jun 15, 2026

linoytsaban and others added 5 commits June 15, 2026 15:59

Merge branch 'main' into ideogram4-lora-loader

611b1ef

document attention_kwargs in Ideogram4 forward and pipeline

77350d9

style Ideogram4 attention_kwargs docstrings

8b202b2

Merge branch 'main' into ideogram4-lora-loader

3e6e6f3

Merge branch 'main' into ideogram4-lora-loader

7e73996

sayakpaul reviewed Jun 17, 2026

View reviewed changes

sayakpaul approved these changes Jun 17, 2026

View reviewed changes

sayakpaul requested a review from dg845 June 17, 2026 02:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Ideogram4LoraLoaderMixin (LoRA loading for Ideogram4)#13921

Add Ideogram4LoraLoaderMixin (LoRA loading for Ideogram4)#13921
linoytsaban wants to merge 12 commits into
mainfrom
ideogram4-lora-loader

linoytsaban commented Jun 11, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Jun 11, 2026

Uh oh!

sayakpaul left a comment

Uh oh!

github-actions Bot commented Jun 13, 2026

Uh oh!

sayakpaul Jun 15, 2026

Uh oh!

sayakpaul left a comment •

edited

Loading

Uh oh!

linoytsaban commented Jun 16, 2026

Uh oh!

sayakpaul Jun 17, 2026

Uh oh!

sayakpaul Jun 17, 2026

Uh oh!

sayakpaul left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

linoytsaban commented Jun 11, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Jun 11, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 13, 2026

Uh oh!

sayakpaul Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linoytsaban commented Jun 16, 2026

Uh oh!

sayakpaul Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sayakpaul left a comment •

edited

Loading