Multimodal Models: Text, Image, Audio and Video
A multimodal model processes more than one type of data, such as text and images, in one request. What it can do, how to design its inputs and where it fails.
What is a multimodal model?
Most LLMs started with text. A multimodal model extends the input: it converts each modality into a representation the model can combine with the rest. For an image, that representation looks like text tokens but comes from a different source. The model does not see pixels directly; it receives an encoding of them.
Integrating several data types is a broad field of deep learning. A 2024 academic survey (Li and Tang) frames it as two linked problems. Alignment means matching modalities so they can be combined, for example knowing which area of an image a phrase describes. Fusion means merging information from several modalities into a single representation. Without good alignment, fusion can produce misreadings or lose information.
Audio and video work differently depending on the product. Some models receive audio directly; others depend on a prior transcript (see AI transcription) or on frames extracted from the video. The difference matters: with a transcript you lose tone, pauses and who spoke. Before designing, confirm in the model’s documentation which modalities it really receives and which are converted to text along the way.
A useful precedent for understanding images is CLIP, presented by OpenAI in 2021. It was trained on about 400 million image and text pairs collected from the internet, with the task of predicting which caption goes with which image. The authors state that after training, the model can classify images using natural language without task-specific examples for each category.
Why does this matter to a designer?
Because multimodality changes the shape of the input the user can give:
- Screenshots as queries. A user can paste a screen with an error and ask what is happening. That reduces the effort of describing the problem in words.
- Sketches and photos. A designer can upload a sketch and ask for a list of components. The result depends on how clear the sketch is.
- New failure modes. The model can misread small text, miscount objects or ignore something at the edge of an image. The interface has to let people correct the reading.
- Privacy. An image can contain personal data the user did not notice. Before sending it, show what will be processed.
Where does it fail today?
Anthropic’s vision documentation is explicit about Claude’s limits. It works as a guide even if you use another model, and it is worth reading each provider’s limitations section, such as the one in OpenAI’s images guide:
| Situation | What the documentation warns | What to do in the interface |
|---|---|---|
| Low-quality, rotated or under-200-pixel images | The model may hallucinate or make mistakes | Warn when the image is small or tilted and ask for another |
| Spatial relationships | Spatial reasoning is limited | Do not base layout decisions on “to the left of” |
| Counting | Quantities may be approximate | Show the number as an estimate |
| AI-generated images | It cannot tell whether an image was AI-generated | Do not use it as a detector |
| Medical diagnostic scans | It is not designed to interpret them | Block or warn against that use |
A wrong reading stated with confidence is a visual hallucination, and you design for it the same way as text ones: show the evidence and let people correct it. To decide how much confidence to convey, see showing uncertainty. These limits change with every model version, so they do not replace your own testing.
When NOT to use it?
When text is enough. Describing an image with AI for something a person can type in ten seconds adds cost and latency. It is also a poor fit for tasks that demand precise measurement, such as exact dimensions of an element in a screenshot; a measuring tool is more reliable there.
Be wary as well when the image is low quality or dense with text. The model can answer with confidence about something it did not read correctly.
How do you design the input?
A few concrete decisions:
- Allow uploading images and also describing what you want, because the combination usually works better than either alone.
- Show the image as the model received it, with the regions it used if possible.
- Offer a field to correct the reading before trusting the answer.
- Limit file size and format, and explain the reason when something is rejected.
- Warn before sending if the image may contain personal data; the topic is developed in privacy in AI products.
- If the image feeds a description for accessibility, review the result by hand; see AI alt text.
For a broader view of how the model uses what it receives, see the context window, since images also take up space in it.
How to test it before launch
- Collect real images from your users, not just clean ones: blurry screenshots, tilted photos, screens with gray text.
- Write the correct answer for each one in advance.
- Compare the model’s answer with the expected one and note the type of error: misread text, counting, position or omission.
- Repeat the test whenever you change model or version, because results do not carry over.
Example
Illustrative example: a QA team uploads a screenshot of a checkout screen and asks why the “Pay” button is disabled. The model identifies that the postal code field is empty and says so. But in another screenshot, with low-contrast gray text, it confuses an error message with an informational notice. The team adds a view where the user sees the text the model extracted and can correct it before the answer is used to decide a design change.
Common mistakes
- Assuming everything the model describes in an image is correct.
- Uploading images with personal data without telling the user.
- Expecting exact measurements from an image.
- Using multimodality for tasks better handled with structured text.
- Testing only with clean, high-resolution images.