Diffusion Models
Diffusion models represent the cutting edge of generative AI, producing some of the most remarkable image synthesis results we've seen to date. Their approach is conceptually beautiful: rather than trying to learn the complex distribution of natural images directly, they learn to gradually remove noise from a pure noise distribution.
The process works in two phases. First, during the forward diffusion process, small amounts of Gaussian noise are gradually added to training images across multiple steps until they become pure noise. Then, a neural network is trained to reverse this process—predicting the noise that was added at each step so it can be removed. This approach transforms the complex problem of generating realistic images into a series of simpler denoising steps.
What makes diffusion models particularly powerful is their flexibility in conditioning. By incorporating text embeddings from large language models, systems like DALL-E, Stable Diffusion, and Midjourney can generate images from detailed text descriptions. This text-to-image capability has democratized visual creation, allowing anyone to generate stunning imagery from natural language prompts.
Beyond their impressive image generation capabilities, diffusion models have shown promise across multiple domains. They excel at image editing tasks like inpainting (filling in missing parts), outpainting (extending images beyond their boundaries), and style transfer. Researchers have adapted the diffusion framework to generate 3D models, video, audio, and even molecular structures for drug discovery.
The theoretical connections between diffusion models and other approaches like score-based generative models and normalizing flows highlight how different perspectives in machine learning can converge on similar solutions. Their success demonstrates that sometimes approaching a problem indirectly—learning to denoise rather than directly generate—can lead to breakthrough results.
Stable Diffusion represents a landmark implementation of the diffusion model approach that balances computational efficiency with generation quality. Unlike earlier diffusion models that operated in pixel space, Stable Diffusion performs the diffusion process in the latent space of a pre-trained autoencoder, dramatically reducing computational requirements while maintaining image quality.
The architecture consists of three main components working in concert. First, a text encoder (typically CLIP) transforms natural language prompts into embedding vectors that guide the generation process. Second, a U-Net backbone serves as the denoising network, progressively removing noise from the latent representation. Finally, a decoder transforms the denoised latent representation back into pixel space to produce the final image.
This design allows Stable Diffusion to generate high-resolution images (typically 512×512 pixels or higher) on consumer GPUs with reasonable memory requirements. The open-source release of the model in 2022 represented a pivotal moment in democratizing access to powerful generative AI, enabling widespread experimentation, fine-tuning for specialized applications, and integration into countless creative tools.
The architecture's flexibility has led to numerous extensions. Techniques like ControlNet add additional conditioning beyond text, allowing image generation to be guided by sketches, pose information, or semantic segmentation maps. LoRA (Low-Rank Adaptation) enables efficient fine-tuning to capture specific styles or subjects with minimal computational resources. Textual inversion methods let users define custom concepts with just a few example images.
This combination of architectural efficiency, powerful generative capabilities, and extensibility has made Stable Diffusion the foundation for an entire ecosystem of image generation applications, from professional creative tools to consumer apps that have introduced millions to the potential of generative AI.
Stable Diffusion's open release in 2022 did something no closed model could: it let an entire research and hobbyist community iterate on the architecture in public. The result is an ecosystem of base models, adaptation techniques and tooling that remains the reference implementation for applied diffusion work, and the place most new conditioning and fine-tuning methods appear first.
The material below is the practitioner's view of that ecosystem — which base models exist, how they are adapted, how they are conditioned, and how they are run. It complements the architectural treatment in the previous section.
Open-weight text-to-image development has passed through several distinct generations, each changing the text encoder, the backbone, or both. Understanding which generation a checkpoint belongs to matters practically, because adapters, conditioning modules and training scripts are generation-specific and rarely transfer.
The Stable Diffusion line
- SD 1.x — 860M-parameter U-Net with a CLIP ViT-L text encoder at 512×512. Architecturally modest, but by far the most heavily tooled checkpoint ever released; the overwhelming majority of community adapters target it.
- SD 2.x — swapped CLIP for OpenCLIP ViT-H and moved to 768×768. Better text alignment, but the change of text encoder invalidated the existing adapter ecosystem, and the release was poorly received as a result — an instructive case of ecosystem lock-in outweighing raw capability.
- SDXL — a 2.6B-parameter U-Net with two text encoders concatenated, a separate refiner stage, and explicit conditioning on original image size and crop coordinates to counteract training-crop artefacts. The first open model competitive with commercial systems on composition.
- SD 3.x — replaced the U-Net with a Multimodal Diffusion Transformer (MMDiT) trained with rectified-flow matching rather than the original DDPM formulation, and used three text encoders including T5-XXL. The T5 encoder is what produced the step-change in prompt adherence and text rendering.
Beyond Stable Diffusion. By 2026 the open-weight frontier has broadened well past Stability AI:
- FLUX (Black Forest Labs) — a rectified-flow transformer from members of the original Stable Diffusion team. The FLUX.2 generation leads open models on realism, texture and native resolution; the smaller distilled variants ship under permissive licences, which is a substantive difference for commercial deployment.
- Qwen-Image (Alibaba) — the strongest open model for text rendering, including CJK scripts where CLIP-derived encoders fail badly.
- HunyuanImage (Tencent) and Sana (NVIDIA) — the latter notable for a deep-compression autoencoder and linear attention that make high-resolution inference viable on modest hardware.
Why generation matters in practice. A LoRA trained for SD 1.5 will not load into SDXL; a ControlNet for SDXL will not work with a FLUX transformer. Adapter compatibility is bounded by the backbone and the text encoder, and it is the single most common source of confusion for people entering the ecosystem.
Fully fine-tuning a multi-billion-parameter diffusion model is expensive and produces an artefact the size of the original checkpoint. The ecosystem instead converged on parameter-efficient adaptation: train a small number of additional parameters that modify the frozen base model's behaviour.
LoRA (Low-Rank Adaptation) is the dominant method. The insight is that the weight update needed to specialise a model is empirically low-rank, so instead of learning a full update matrix ΔW for a layer, you learn two thin matrices A and B and use their product:
W' = W + BA where W ∈ R^(d×k), B ∈ R^(d×r), A ∈ R^(r×k), r << min(d,k)With rank r typically between 4 and 128, this trains a fraction of a percent of the parameters. In diffusion models the adapters are usually applied to the cross-attention projections — the layers where text conditioning enters — which is precisely where a style or subject needs to be injected.
Practical consequences of the low-rank formulation:
- Adapter files are megabytes rather than gigabytes, so they are trivially shareable
- Training runs on a single consumer GPU in minutes to hours
- Because the update is additive, multiple adapters compose at inference time with per-adapter scaling weights — though interference between adapters trained on overlapping concepts is common
- Adapters can be merged permanently into the base weights, trading composability for inference speed
Variants: LyCORIS generalises the decomposition beyond simple low-rank products (LoHa uses Hadamard products, LoKr Kronecker products) for higher expressive capacity at similar parameter counts. DoRA decomposes the update into separate magnitude and direction components, which improves stability at low rank and narrows the gap to full fine-tuning.
Other adaptation approaches:
- Textual Inversion — freezes the entire model and learns only a new token embedding for a concept. Kilobytes rather than megabytes, but limited to what the frozen model can already express.
- DreamBooth — full fine-tuning on a handful of subject images, using a rare identifier token and a prior-preservation loss to avoid language drift and catastrophic forgetting of the broader class. Highest subject fidelity, heaviest cost; commonly combined with LoRA to get most of the quality at a fraction of the compute.
- Hypernetworks — a small auxiliary network that generates modifications to attention layers at inference time. Largely superseded by LoRA.
- Quantisation — post-training reduction of weight precision to 8- or 4-bit, plus techniques such as NF4. Not adaptation but deployment: it is what makes multi-billion-parameter models fit in consumer VRAM, at some cost in fine detail.
Text conditioning alone gives no control over spatial structure. A family of techniques adds further conditioning signals to the denoising process, and they are what make diffusion models usable in production pipelines rather than only as samplers.
ControlNet clones the encoder half of the denoising backbone into a trainable copy, freezes the original, and connects the copy back into the frozen decoder through zero-initialised convolutions. Because those connections start at zero, the augmented model is initially identical to the base model and degrades gracefully during training rather than destroying learned behaviour.
The trainable copy receives a spatial conditioning map — Canny edges, a depth estimate, an OpenPose skeleton, surface normals, semantic segmentation, a scribble — and injects structural guidance at every resolution of the decoder. Multiple ControlNets can be stacked with independent weights.
Related conditioning mechanisms:
- T2I-Adapter — a lighter alternative that trains small feature extractors injected into the encoder, with far fewer parameters than ControlNet at some cost in fidelity.
- IP-Adapter — adds a decoupled cross-attention pathway for image prompts, so visual and textual conditioning are attended separately rather than concatenated. This is the mechanism behind most style- and face-reference features in commercial products.
- Image-to-image (SDEdit) — rather than starting from pure noise, noise the input image to an intermediate timestep and denoise from there. The chosen timestep is the strength parameter: it sets how much of the input's structure survives.
- Inpainting — at each denoising step, replace the unmasked region with a correspondingly-noised version of the original. The model only ever generates inside the mask while attending to the true surrounding context.
- Classifier-free guidance — the underlying knob behind every "guidance scale" or "CFG" setting. Each step runs the model twice, conditioned and unconditioned, and extrapolates along their difference:
ε = ε_uncond + s·(ε_cond − ε_uncond). High s improves prompt adherence and saturates colour and contrast; the negative prompt is simply a non-empty unconditional branch.
Applied diffusion work is dominated by a small number of inference frontends. The distinction that matters is between fixed-pipeline interfaces, which expose a parameter form over a predetermined graph, and node-based systems, which expose the computation graph itself.
ComfyUI
A node-based frontend that exposes the diffusion pipeline as an explicit directed graph — model loading, conditioning, sampling, latent operations and decoding are all separate, rewireable nodes. This makes non-standard pipelines (multi-pass sampling, region-specific conditioning, mixed models, video workflows) expressible without code, and it is why new research techniques almost always ship as ComfyUI nodes first. Now the de facto standard for serious work, at the cost of a steep initial learning curve.
Automatic1111 WebUI
The original mass-adoption interface, and still the most direct route from checkpoint to image. A fixed pipeline with an extensive parameter surface and a large extension ecosystem. Development has slowed considerably in favour of the Forge fork, but its API and directory conventions remain a de facto standard that other tools follow.
Forge / Forge Neo
A fork of the Automatic1111 WebUI with a substantially rewritten backend — better memory management, faster sampling, and support for newer architectures including FLUX. Generally the recommended choice over upstream A1111 for anyone who prefers a fixed-pipeline interface.
InvokeAI
A production-oriented frontend with a unified canvas that treats generation, inpainting and outpainting as operations on a single workspace rather than separate modes. Also ships a node editor. The most coherent option for iterative compositing workflows.
Diffusers (Hugging Face)
The reference Python library rather than an interface — modular pipelines, schedulers and model classes covering essentially every published diffusion architecture. The right layer for programmatic work, custom research, and anything that needs to run as a service rather than as an application.
SD.Next
A heavily refactored fork emphasising broad backend support — CUDA, ROCm, Intel XPU, Apple MPS, DirectML — and rapid adoption of new model architectures. The pragmatic choice on non-NVIDIA hardware.
Fooocus
A deliberately minimal interface that hides nearly every parameter behind curated defaults and automatic prompt expansion. Useful as a baseline for what a well-tuned default pipeline produces, and as a low-friction deployment for non-technical users.
DiffusionBee
A self-contained macOS application built on Apple Silicon's unified memory architecture, requiring no Python environment. The simplest possible local deployment on a Mac.
Deployment constraints. VRAM, not compute, is the binding limit for local inference. Roughly: SD 1.5 runs comfortably in 4GB, SDXL wants 8–12GB, and FLUX-class transformer models need 16–24GB at full precision — or considerably less with 8- and 4-bit quantisation, attention slicing, sequential CPU offloading, and VAE tiling for high-resolution decoding. Where local hardware is insufficient, managed platforms (Replicate, RunDiffusion, fal, Hugging Face Inference Endpoints) offer the same models per-second, and several provide one-click ComfyUI or A1111 instances.
Extending diffusion to three dimensions runs into an immediate data problem: there is no 3D corpus remotely comparable in scale to the image-text pairs that made 2D generation work. The dominant research response has been to avoid training on 3D data at all, and instead distil 3D structure out of a pretrained 2D model.
Score Distillation Sampling (SDS), introduced with DreamFusion, is the key idea. A 3D representation — typically a NeRF — is optimised so that renders from randomly sampled camera angles all look, to a frozen 2D diffusion model, like plausible samples for the text prompt. The 2D model's denoising prediction is used directly as a gradient on the 3D parameters; no 3D training data is required.
The characteristic failure modes follow from the method: the Janus problem (a face on every side, because every viewpoint is independently pushed toward the canonical view of the prompt), over-saturated colours from the high guidance scales SDS requires, and slow per-asset optimisation. Subsequent work — Magic3D's coarse-to-fine two-stage pipeline, variational reformulations of the objective, and multi-view-consistent 2D priors — addresses these to varying degrees.
Direct 3D generation takes the opposite approach, training generative models on 3D data where it exists:
- Point-E — a two-stage system generating a synthetic view with a 2D model, then a point cloud conditioned on it. Orders of magnitude faster than optimisation-based methods, at much lower fidelity.
- Shap-E — generates parameters of an implicit function directly, yielding both a signed distance field and a texture field, and so producing meshes and radiance fields rather than point clouds.
- GET3D — a GAN producing explicit textured meshes with arbitrary topology via a differentiable surface extraction, designed for direct use in graphics pipelines.
- Native 3D diffusion (Hunyuan3D, Trellis and successors) — trained on large curated 3D asset collections, generating latent shape representations that decode to meshes. This is the line that has produced practically usable output, and it now underpins most commercial 3D generation services.
Neural scene representations are the substrate much of this builds on, and are significant independently of generation:
- NeRF represents a scene as a continuous function mapping position and viewing direction to colour and volume density, trained by differentiable volume rendering against posed input images. Photorealistic novel-view synthesis, but slow to train and slower to render, since every pixel requires many network evaluations along a ray.
- 3D Gaussian Splatting replaces the implicit field with an explicit set of anisotropic 3D Gaussians, each with position, covariance, opacity and spherical-harmonic colour, rasterised by a differentiable tile-based renderer. Comparable or better quality than NeRF with training in minutes and rendering in real time — the change that moved neural scene representation from research into production, and the reason phone-based 3D capture became viable.