Knowledge Labs
Logo
Introduction to Machine LearningDeep Learning IntroductionProbability for Machine LearningStatistics for Machine LearningNatural Language Processing (NLP)Data ScienceAI for EveryoneRetrieval Augmented Generation (RAG)AI for ArtistsModel Context ProtocolComprehensive Guide to Economy & Financial SystemsEconomic CyclesMoney & Central BankingBonds & the Yield CurveResources & WorkProbability & PredictionIntroduction to Babylon.js: A Powerful Web Game Engine
Deep Learning Introduction
/Neural Network Architectures
/Diffusion Models
/The Open-Weight Diffusion Ecosystem
/Fine-Tuning and Adaptation

Fine-Tuning and Adaptation

.

Fine-Tuning and Adaptation

Fully fine-tuning a multi-billion-parameter diffusion model is expensive and produces an artefact the size of the original checkpoint. The ecosystem instead converged on parameter-efficient adaptation: train a small number of additional parameters that modify the frozen base model's behaviour.

LoRA (Low-Rank Adaptation) is the dominant method. The insight is that the weight update needed to specialise a model is empirically low-rank, so instead of learning a full update matrix ΔW for a layer, you learn two thin matrices A and B and use their product:

W' = W + BA        where  W ∈ R^(d×k),  B ∈ R^(d×r),  A ∈ R^(r×k),  r << min(d,k)

With rank r typically between 4 and 128, this trains a fraction of a percent of the parameters. In diffusion models the adapters are usually applied to the cross-attention projections — the layers where text conditioning enters — which is precisely where a style or subject needs to be injected.

Practical consequences of the low-rank formulation:

  • Adapter files are megabytes rather than gigabytes, so they are trivially shareable
  • Training runs on a single consumer GPU in minutes to hours
  • Because the update is additive, multiple adapters compose at inference time with per-adapter scaling weights — though interference between adapters trained on overlapping concepts is common
  • Adapters can be merged permanently into the base weights, trading composability for inference speed

Variants: LyCORIS generalises the decomposition beyond simple low-rank products (LoHa uses Hadamard products, LoKr Kronecker products) for higher expressive capacity at similar parameter counts. DoRA decomposes the update into separate magnitude and direction components, which improves stability at low rank and narrows the gap to full fine-tuning.

Other adaptation approaches:

  • Textual Inversion — freezes the entire model and learns only a new token embedding for a concept. Kilobytes rather than megabytes, but limited to what the frozen model can already express.
  • DreamBooth — full fine-tuning on a handful of subject images, using a rare identifier token and a prior-preservation loss to avoid language drift and catastrophic forgetting of the broader class. Highest subject fidelity, heaviest cost; commonly combined with LoRA to get most of the quality at a fraction of the compute.
  • Hypernetworks — a small auxiliary network that generates modifications to attention layers at inference time. Largely superseded by LoRA.
  • Quantisation — post-training reduction of weight precision to 8- or 4-bit, plus techniques such as NF4. Not adaptation but deployment: it is what makes multi-billion-parameter models fit in consumer VRAM, at some cost in fine detail.