Knowledge Labs
Logo
Introduction to Machine LearningDeep Learning IntroductionProbability for Machine LearningStatistics for Machine LearningNatural Language Processing (NLP)Data ScienceAI for EveryoneRetrieval Augmented Generation (RAG)AI for ArtistsModel Context ProtocolComprehensive Guide to Economy & Financial SystemsEconomic CyclesMoney & Central BankingBonds & the Yield CurveResources & WorkProbability & PredictionIntroduction to Babylon.js: A Powerful Web Game Engine
Deep Learning Introduction
/Neural Network Architectures
/Diffusion Models
/Diffusion Beyond Images: 3D and Neural Scene Representation

Diffusion Beyond Images: 3D and Neural Scene Representation

.

Diffusion Beyond Images: 3D and Neural Scene Representation

Extending diffusion to three dimensions runs into an immediate data problem: there is no 3D corpus remotely comparable in scale to the image-text pairs that made 2D generation work. The dominant research response has been to avoid training on 3D data at all, and instead distil 3D structure out of a pretrained 2D model.

Score Distillation Sampling (SDS), introduced with DreamFusion, is the key idea. A 3D representation — typically a NeRF — is optimised so that renders from randomly sampled camera angles all look, to a frozen 2D diffusion model, like plausible samples for the text prompt. The 2D model's denoising prediction is used directly as a gradient on the 3D parameters; no 3D training data is required.

The characteristic failure modes follow from the method: the Janus problem (a face on every side, because every viewpoint is independently pushed toward the canonical view of the prompt), over-saturated colours from the high guidance scales SDS requires, and slow per-asset optimisation. Subsequent work — Magic3D's coarse-to-fine two-stage pipeline, variational reformulations of the objective, and multi-view-consistent 2D priors — addresses these to varying degrees.

Direct 3D generation takes the opposite approach, training generative models on 3D data where it exists:

  • Point-E — a two-stage system generating a synthetic view with a 2D model, then a point cloud conditioned on it. Orders of magnitude faster than optimisation-based methods, at much lower fidelity.
  • Shap-E — generates parameters of an implicit function directly, yielding both a signed distance field and a texture field, and so producing meshes and radiance fields rather than point clouds.
  • GET3D — a GAN producing explicit textured meshes with arbitrary topology via a differentiable surface extraction, designed for direct use in graphics pipelines.
  • Native 3D diffusion (Hunyuan3D, Trellis and successors) — trained on large curated 3D asset collections, generating latent shape representations that decode to meshes. This is the line that has produced practically usable output, and it now underpins most commercial 3D generation services.

Neural scene representations are the substrate much of this builds on, and are significant independently of generation:

  • NeRF represents a scene as a continuous function mapping position and viewing direction to colour and volume density, trained by differentiable volume rendering against posed input images. Photorealistic novel-view synthesis, but slow to train and slower to render, since every pixel requires many network evaluations along a ray.
  • 3D Gaussian Splatting replaces the implicit field with an explicit set of anisotropic 3D Gaussians, each with position, covariance, opacity and spherical-harmonic colour, rasterised by a differentiable tile-based renderer. Comparable or better quality than NeRF with training in minutes and rendering in real time — the change that moved neural scene representation from research into production, and the reason phone-based 3D capture became viable.