Unconditional Image Generation with provided noisy laten image or Image2Image without prompt

[youyinnn]

Hi there.

I am looking for how to generate an image with an unconditional diffusion manner that is able to provide a noisy image.
Or, as an image task but without a prompt, the prompt requires a trained domain-specific NLP model, I suppose (but it's not available in our case).

Any ideas?

[youyinnn]

Also, what about the custom scheduler? I want to customize the noise generation during the prediction, not the training. Is there a way to do that?

[tolgacangoz]

Hi @youyinnn,
Could you examine an unconditional image generation pipeline's code to directly reason about it? Do you want to provide the latents variable on your own?
By saying Image2Image without a prompt, are you referring to image variations?

[youyinnn]

Hello, thanks for the resources. I think what I mean is more of an image variation or reconstruction. The only purpose is to give the model a corrupted or masked image, and I want the model to generate a variation based on that.

I just had a quick look at the two links you provided. For the first one, I think I can try to have the actual sample here laten and have my own scheduler (to generate the custom noise), right?

The image variation is also similar to what I need.

Thank you again.

[y27390641-sudo]

Интересная статья, особенно полезна часть про прямой процесс и добавление шума. Насколько я понимаю, для безусловной генерации текстовый промпт вообще не обязателен — модель может начинать с обычного шума и постепенно восстанавливать изображение, если она обучена на нужном домене. Для быстрого сравнения современных вариантов генерации изображений можно также посмотреть Ranvik, а уже для собственного эксперимента разбираться с scheduler и параметрами шума. Думаю, такой подход удобнее, если задача именно в работе с зашумлёнными изображениями, а не в генерации по текстовому описанию.

[tk1475]

For no prompt you don't need a special model. Pass an empty prompt and turn off guidance:

from diffusers import StableDiffusionImg2ImgPipeline
import torch

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")

out = pipe(prompt="", image=corrupted, strength=0.5, guidance_scale=1.0).images[0]

guidance_scale of 1.0 makes it fully unconditional, and strength sets how far it moves from your input (low stays close, high varies more). That covers variation and reconstruction from a corrupted image.

Two variants depending on what you have:

• If you have a mask for the missing region, use the inpaint pipeline instead. It only fills the masked area and leaves the rest untouched.
• To control the noise yourself, both img2img and the base pipelines accept a latents argument, so you can build your own noised latent and pass it in. That's your hook for a custom scheduler, no subclassing needed.

And if you want a truly unconditional pixel model with no text encoder at all, look at DDPMPipeline or DDIMPipeline with something like google/ddpm-celebahq-256. It's the cleanest example to read since there's no prompt path to work around.