Multimodal
Capable of processing and generating multiple types of data — such as text, images, audio, and video — within a single model.
A multimodal AI model can understand and reason across different modalities simultaneously. Rather than having separate models for vision and language, multimodal models like GPT-4o, Gemini, and Claude 3.5 Sonnet can analyze an image while discussing it in text, understand audio alongside a transcript, or generate code based on a screenshot.
Most current multimodal LLMs are primarily text models with additional encoder components (like CLIP for images) that project other modalities into the model's text embedding space. True native multimodality — where the model processes all modalities with the same architecture from training — is an active research frontier, with Google's Gemini and Meta's Llama 4 being early examples.
Multimodal capabilities unlock a wide range of applications: analyzing medical imaging alongside patient records, processing handwritten notes, understanding complex charts and diagrams, building visual coding assistants that can see a UI mockup and generate the code, and creating accessibility tools that describe images for visually impaired users.
Read next
Multimodal LLMs: AI That Sees, Hears, and Reads
How modern AI models process multiple modalities — text, images, audio, and video simultaneously — and what this enables for real-world applications.
What Is a Large Language Model?
A clear, jargon-free introduction to large language models — what they are, how they work at a high level, and why they're transforming software and society.
OpenAI: The Lab That Started the AI Revolution
The complete story of OpenAI — from its nonprofit founding to GPT-5, ChatGPT, and the o-series reasoning models that defined the AI era.