Multimodal RAG
Generate answers from text and visual evidence, choosing between descriptions and vision-language models by cost and need.
Answers built from text and pictures
Multimodal RAG retrieves evidence of different kinds, such as passages, tables, charts and page images, and gives it to a model that can read all of them to write the answer. The question "How did placements change between 2023 and 2025, and which company hired the most?" may need a bar chart (for the trend) and a table (for the company counts). A text-only system can answer only if parsing captured both perfectly. A multimodal system can look at the chart and the table directly.
Two ways to generate
There are two broad designs for the answer step.
Convert everything to text first
Retrieved images are turned into text descriptions (or were already described during ingestion), and an ordinary text model writes the answer. This is cheaper and works with any language model, but the answer is only as good as the descriptions.
Give images directly to a vision-language model
The retrieved page images or figures are placed in the prompt alongside the text passages, and a model that accepts images reads them directly. This keeps every detail of the original, at a higher cost per question, because images use many tokens.
Converting to text is cheaper; reading images directly preserves more detail.
Building a multimodal prompt
A multimodal prompt looks much like a text RAG prompt, with images added as numbered sources:
Answer using ONLY the sources. Cite them as [1], [2]. Read charts and tables carefully,
and say if a value cannot be read clearly.
[1] Annual Report 2025, p.12 — image: bar chart "Students placed per year"
[2] Annual Report 2025, p.13 — table: company-wise placements (Markdown)
[3] Annual Report 2025, p.12 — text: "Placements grew strongly over three years."
Question: How did placements change from 2023 to 2025, and which company hired the most?
The instruction to say when a value cannot be read clearly matters: vision-language models can misread small or crowded charts just as OCR misreads characters, and it is far better for the answer to admit uncertainty than to state a wrong number confidently.
Citations that point to the picture
Multimodal answers should cite visual evidence as precisely as text: the document, the page, and ideally the region of the page (its bounding box) or the figure number. The user interface can then show the actual chart or highlight the relevant area on the page image next to the answer, which lets the user check the reading for themselves.
Compare three ways of answering one question below.
The cost of images in the prompt
Images are converted into many tokens inside the model, often hundreds to over a thousand per image depending on the model and the image size. A prompt with five page images can easily cost more than one with fifty text passages. Use the calculator to estimate.
Practical habits keep costs down: send only the top one or two images, crop to the relevant figure rather than whole pages where possible, and use text descriptions for routine questions and images only when the question is about a visual.
The program below sketches the decision of when to include images, based on what the retrieved evidence contains.
Expected output:
[1] Students placed per year (chart) -> send the IMAGE to a vision-language model
[2] Company-wise placements -> send its text description / Markdown
[3] Placements paragraph -> send the text
needs_visual_detail is a simple keyword test on the question. The if ladder then decides, for each retrieved item, the cheapest form that will still answer it: the image itself only when the item is a figure and the question needs to read detail from it, a text description or Markdown for other figures and tables, and plain text for everything else.
Where multimodal RAG pays off
Multimodal RAG is worth its extra cost and complexity when a meaningful share of questions depend on visuals: financial and annual reports, engineering manuals with diagrams, scientific papers, medical images with reports, product catalogues, slide decks and maps. For collections that are mostly prose, a text pipeline with good figure descriptions is usually enough.