Knowledge Labs
Logo
Introduction to Machine LearningDeep Learning IntroductionProbability for Machine LearningStatistics for Machine LearningNatural Language Processing (NLP)Data ScienceAI for EveryoneRetrieval Augmented Generation (RAG)AI for ArtistsModel Context ProtocolComprehensive Guide to Economy & Financial SystemsEconomic CyclesMoney & Central BankingBonds & the Yield CurveResources & WorkProbability & PredictionIntroduction to Babylon.js: A Powerful Web Game Engine
Deep Learning Introduction
/Neural Network Architectures
/Transformers
/Multi-Modal Transformers

Multi-Modal Transformers

.

Multi-Modal Transformers

The architectural convergence of the last few years is that text, images, audio and video can all be tokenised into a shared sequence and processed by a single transformer, rather than routed through separate modality-specific models with a fusion layer bolted on top.

The unifying mechanism is tokenisation. Text becomes subword tokens; an image becomes a grid of patch embeddings; video becomes spatio-temporal patches spanning both space and time; audio becomes discrete acoustic tokens from a learned codec. Once every modality is a sequence of vectors in a shared space, self-attention operates across all of them indiscriminately, and cross-modal relationships are learned by exactly the same mechanism as within-modality ones.

This is why capability transfers across modalities in ways that separately-trained systems never managed. A model that has learned spatial and physical relations from text can apply that structure to images; a model that has seen images can ground language in perceptual reality rather than only in co-occurrence statistics.

Architectural approaches, roughly in order of integration:

  • Late fusion / adapter — a frozen vision encoder feeds a projection layer into a frozen language model, with only the connector trained. Cheap and effective (the LLaVA family), but the modalities never truly share representations.
  • Interleaved training — modality-specific encoders with a jointly trained transformer over the combined sequence. Strong cross-modal reasoning, though generation typically still requires a separate decoder head.
  • Natively multimodal — trained from scratch on interleaved data across all modalities, with unified input and output. Gemini and GPT-4o-class models take this path, and it is what enables genuinely low-latency conversational interaction over images and audio.

Consequences worth noting. Unified architectures collapse a pipeline of specialised models into one, which simplifies deployment but concentrates capability and cost. Emergent behaviours appear that were not explicitly trained for — reading a chart and reasoning about it, describing an image in one language having been asked in another, following an instruction that only makes sense given both an image and a sentence. And the same architecture scales down: distilled on-device variants bring multimodal understanding to phones with acceptable latency and no network round-trip, which is a meaningful privacy property rather than only an engineering one.