Conditioning and Structural Control
Text conditioning alone gives no control over spatial structure. A family of techniques adds further conditioning signals to the denoising process, and they are what make diffusion models usable in production pipelines rather than only as samplers.
ControlNet clones the encoder half of the denoising backbone into a trainable copy, freezes the original, and connects the copy back into the frozen decoder through zero-initialised convolutions. Because those connections start at zero, the augmented model is initially identical to the base model and degrades gracefully during training rather than destroying learned behaviour.
The trainable copy receives a spatial conditioning map — Canny edges, a depth estimate, an OpenPose skeleton, surface normals, semantic segmentation, a scribble — and injects structural guidance at every resolution of the decoder. Multiple ControlNets can be stacked with independent weights.
Related conditioning mechanisms:
- T2I-Adapter — a lighter alternative that trains small feature extractors injected into the encoder, with far fewer parameters than ControlNet at some cost in fidelity.
- IP-Adapter — adds a decoupled cross-attention pathway for image prompts, so visual and textual conditioning are attended separately rather than concatenated. This is the mechanism behind most style- and face-reference features in commercial products.
- Image-to-image (SDEdit) — rather than starting from pure noise, noise the input image to an intermediate timestep and denoise from there. The chosen timestep is the strength parameter: it sets how much of the input's structure survives.
- Inpainting — at each denoising step, replace the unmasked region with a correspondingly-noised version of the original. The model only ever generates inside the mask while attending to the true surrounding context.
- Classifier-free guidance — the underlying knob behind every "guidance scale" or "CFG" setting. Each step runs the model twice, conditioned and unconditioned, and extrapolates along their difference:
ε = ε_uncond + s·(ε_cond − ε_uncond). High s improves prompt adherence and saturates colour and contrast; the negative prompt is simply a non-empty unconditional branch.