Diffusion Models
Diffusion models represent the cutting edge of generative AI, producing some of the most remarkable image synthesis results we've seen to date. Their approach is conceptually beautiful: rather than trying to learn the complex distribution of natural images directly, they learn to gradually remove noise from a pure noise distribution.
The process works in two phases. First, during the forward diffusion process, small amounts of Gaussian noise are gradually added to training images across multiple steps until they become pure noise. Then, a neural network is trained to reverse this process—predicting the noise that was added at each step so it can be removed. This approach transforms the complex problem of generating realistic images into a series of simpler denoising steps.
What makes diffusion models particularly powerful is their flexibility in conditioning. By incorporating text embeddings from large language models, systems like DALL-E, Stable Diffusion, and Midjourney can generate images from detailed text descriptions. This text-to-image capability has democratized visual creation, allowing anyone to generate stunning imagery from natural language prompts.
Beyond their impressive image generation capabilities, diffusion models have shown promise across multiple domains. They excel at image editing tasks like inpainting (filling in missing parts), outpainting (extending images beyond their boundaries), and style transfer. Researchers have adapted the diffusion framework to generate 3D models, video, audio, and even molecular structures for drug discovery.
The open-weight side of diffusion has grown into an ecosystem of its own — base model generations and what separates them, LoRA and the other parameter-efficient adaptations, ControlNet conditioning, and the inference frontends people actually run them in. That practitioner material has its own article: Open-Weight Diffusion in Practice.
Stable Diffusion represents a landmark implementation of the diffusion model approach that balances computational efficiency with generation quality. Unlike earlier diffusion models that operated in pixel space, Stable Diffusion performs the diffusion process in the latent space of a pre-trained autoencoder, dramatically reducing computational requirements while maintaining image quality.
The architecture consists of three main components working in concert. First, a text encoder (typically CLIP) transforms natural language prompts into embedding vectors that guide the generation process. Second, a U-Net backbone serves as the denoising network, progressively removing noise from the latent representation. Finally, a decoder transforms the denoised latent representation back into pixel space to produce the final image.
This design allows Stable Diffusion to generate high-resolution images (typically 512×512 pixels or higher) on consumer GPUs with reasonable memory requirements. The open-source release of the model in 2022 represented a pivotal moment in democratizing access to powerful generative AI, enabling widespread experimentation, fine-tuning for specialized applications, and integration into countless creative tools.
The architecture's flexibility has led to numerous extensions — additional conditioning signals beyond text, and cheap ways to fine-tune it for a particular style or subject. Those are covered in Open-Weight Diffusion in Practice.
This combination of architectural efficiency, powerful generative capabilities, and extensibility has made Stable Diffusion the foundation for an entire ecosystem of image generation applications, from professional creative tools to consumer apps that have introduced millions to the potential of generative AI.
Extending diffusion to three dimensions runs into an immediate data problem: there is no 3D corpus remotely comparable in scale to the image-text pairs that made 2D generation work. The dominant research response has been to avoid training on 3D data at all, and instead distil 3D structure out of a pretrained 2D model.
Score Distillation Sampling (SDS), introduced with DreamFusion, is the key idea. A 3D representation — typically a NeRF — is optimised so that renders from randomly sampled camera angles all look, to a frozen 2D diffusion model, like plausible samples for the text prompt. The 2D model's denoising prediction is used directly as a gradient on the 3D parameters; no 3D training data is required.
The characteristic failure modes follow from the method: the Janus problem (a face on every side, because every viewpoint is independently pushed toward the canonical view of the prompt), over-saturated colours from the high guidance scales SDS requires, and slow per-asset optimisation. Subsequent work — Magic3D's coarse-to-fine two-stage pipeline, variational reformulations of the objective, and multi-view-consistent 2D priors — addresses these to varying degrees.
Direct 3D generation takes the opposite approach, training generative models on 3D data where it exists. Early systems traded fidelity for speed: Point-E generated a synthetic view with a 2D model and then a point cloud conditioned on it, Shap-E produced implicit-function parameters yielding meshes and radiance fields rather than point clouds, and GET3D used a GAN to emit textured meshes directly usable in graphics pipelines. What made the approach practical was native 3D diffusion (Hunyuan3D, Trellis and successors) — trained on large curated 3D asset collections, generating latent shape representations that decode to meshes. That line now underpins most commercial 3D generation services.
Neural scene representations are the substrate much of this builds on, and are significant independently of generation:
- NeRF represents a scene as a continuous function mapping position and viewing direction to colour and volume density, trained by differentiable volume rendering against posed input images. Photorealistic novel-view synthesis, but slow to train and slower to render, since every pixel requires many network evaluations along a ray.
- 3D Gaussian Splatting replaces the implicit field with an explicit set of anisotropic 3D Gaussians, each with position, covariance, opacity and spherical-harmonic colour, rasterised by a differentiable tile-based renderer. Comparable or better quality than NeRF with training in minutes and rendering in real time — the change that moved neural scene representation from research into production, and the reason phone-based 3D capture became viable.