03 · Foundation Models & LLMs · Topic 06

What is a Multimodal Model?

A multimodal model can process or generate information across more than one type of data, such as text, images, audio, or video.

What is it?

A multimodal model works with more than one type of information within one model or connected modeling system.

Common modalities include text, image, audio, and video. Some models accept several input modalities; some also generate several output modalities. A multimodal model does not necessarily support every modality.

TextImageAudioVideo

Think of it like...

Think of a person receiving information through several channels instead of only reading text.

For example: read a question + look at an image + produce a text answer.

How it works

1 · InputsReceive informationText, image, audio, or video arrives.

2 · ProcessingHandle each modalityModality-specific processing turns inputs into useful signals.

3 · RepresentationConnect meaningRepresentations are shared or connected across modalities.

4 · ModelReason over the inputThe model processes the combined information.

5 · OutputReturn a resultIt may answer, classify, describe, or generate.

Real-world examples

Image + text question answering

Input: A photo and a question

Output: A text answer about the photo

Speech + text interaction

Input: Spoken words and written context

Output: A response in text or speech

Video understanding

Input: A video with audio

Output: A summary, label, or answer about what happened

What it is NOT

Multimodal Model ≠ LLM

Multimodal Model: Works across more than one data type.

LLM: Primarily built to process and generate language.

Multimodal Model ≠ Generative AI

Multimodal Model: Describes model input and output capability.

Generative AI: Describes systems that create new content.

Multimodal Model ≠ Multimodal AI

Multimodal Model: The model itself.

Multimodal AI: The broader category of AI systems and experiences that work across modalities.

Remember this