recallA retrieval
field guide

Multimodal Enterprise Knowledge

Build this skill

Identify what text-only pipelines lose from charts, tables, images, audio and video, and the two ways to make them searchable.

Knowledge is not only text

Open the Kaveri Institute's annual report. The number of students placed each year is a bar chart. The fee structure is a table. The campus map is an image. The hostel evacuation procedure is a flowchart. The convocation address is a video, and the last academic council meeting exists only as an audio recording. A text-only RAG system sees none of these, or sees them only as captions and scraps of extracted text.

Multimodal means involving more than one kind of data, or modality: text, images, tables, charts, diagrams, slides, audio and video. A multimodal knowledge system can store, search and answer from all of them, not just from paragraphs.

What text-only pipelines lose

Content What a text-only pipeline keeps What it loses
Bar chart of placements The title, maybe axis labels Every value, and the trend
Fee table Numbers, if parsing works; often jumbled Which number belongs to which row and column
Campus map Nothing Where buildings are
Flowchart of a procedure Box labels as loose words The order of steps and the decisions
Scanned signed notice OCR text, with errors Stamps, signatures, layout
Slide deck Bullet fragments Diagrams and the meaning they carry
Lecture or meeting recording Nothing Everything that was said

These are not rare corners of a collection. In reports, manuals and research papers, the most important facts are often in the figures and tables, because that is where authors put the numbers that matter.

Textparagraphs, listsembed directlyTableskeep rows and columnsMarkdown or databaseImages & chartsdescribe or embedwith vision modelsPage imagessearch the page itselflayout preservedAudio & videotranscribe with timestampsthen treat as text

Each modality needs its own way into the index, but they should all be searchable together.

Two ways to make non-text content searchable

There are two broad strategies, and most systems use both.

Convert to text

Turn every non-text item into text that describes it: a chart becomes "Bar chart: students placed rose from 412 in 2023 to 538 in 2025"; a table becomes Markdown; audio becomes a transcript. Then use ordinary text retrieval. This is simple and reuses everything you already have, but the description can miss or misread details.

Search the content itself

Use models that understand images directly. Multimodal embedding models place images and text in the same vector space, so a text question can find an image. Visual document retrieval embeds pictures of whole pages, so layout, charts and tables are searched as they appear, without parsing. This keeps more information, at the cost of new models and larger indexes.

Visually rich documents

Some documents are called visually rich: their meaning depends on layout, tables, charts and images as much as on running text. Annual reports, scientific papers, brochures, forms and slide decks are typical examples. Research on retrieval over such documents, for example the VDocRAG work presented at CVPR 2025, shows that treating pages as images, rather than only extracting their text, can substantially improve retrieval and question answering on these collections.

The program below takes a small inventory of a collection by modality, the same kind of profile that tells you where a multimodal pipeline is needed.

python · runs in your browser

Expected output:

text
items by modality: {'chart': 2, 'table': 2, 'text': 2, 'image': 1, 'diagram': 1, 'video': 1, 'audio': 1}
searchable by a text-only pipeline without extra work: 2 of 10

Each item is a pair of (where it is, what kind of content it is). Counter(kind for _, kind in items) tallies the kinds, and by_kind["text"] reads off how many are plain text.

Only two of ten items are plain text. Everything else needs either conversion or a multimodal model to become searchable.

Questions that need multimodal retrieval

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close