Multimodal Knowledge Base Design
Design a knowledge base that stores documents, pages and linked elements so any hit can be expanded and cited.
Keeping text, tables and images connected
A page of the annual report contains a heading, two paragraphs, a table of company-wise placements and a bar chart. After multimodal ingestion, you have several pieces: text chunks, a Markdown table, a chart image, its caption, a generated description, maybe a page image with patch vectors. If these are stored as unrelated items, two problems appear. A search that finds the chart cannot show the paragraph that explains it, and a search that finds the paragraph cannot show the chart it refers to. The pieces lose each other.
A multimodal knowledge base stores every kind of content in one organised design, where each piece knows which document and page it came from and which other pieces it belongs with. That lets the system search each modality in the way that suits it, and still assemble the right combination of text, tables and images for an answer.
A unified design
A practical design has three layers.
Every element points to its page, and every page to its document, so any search hit can be expanded into its full context.
Documents hold file-level metadata, as in any knowledge base. Pages hold page-level information, including the rendered page image, which is useful both for page-image retrieval and for showing users the original. Elements are the searchable pieces: each has a type (text, table, figure, caption, transcript segment), its content (text, Markdown or a link to an image file), its embedding(s), its position on the page, and links to related elements, such as "this caption belongs to this figure" or "this paragraph refers to Figure 3".
The images themselves are usually stored as files in object storage (a service for storing large files, such as a cloud storage bucket or a shared drive), with only their location kept in the database.
Which embeddings to store
Different elements need different embeddings:
| Element | Embeddings to store |
|---|---|
| Text chunk | Text embedding (and keyword index) |
| Table | Text embedding of the Markdown, plus its caption |
| Figure | Image embedding, plus a text embedding of its caption and generated description |
| Page | Page-image patch vectors, if using page-image retrieval |
| Audio or video segment | Text embedding of the transcript segment, with start and end times |
Storing a text embedding of every figure's caption and description, even when an image embedding exists, is a cheap and very effective habit: it makes figures findable by ordinary text search, and it keeps the system working even when the multimodal model is weak on a particular kind of image.
Linking pieces back together at answer time
When a search finds an element, the system can expand it using the links: a chart hit brings its caption and the paragraph that discusses it; a paragraph hit that mentions "Figure 3" brings the figure. The program below builds this structure in SQLite and shows the expansion step.
Expected output:
Annual Report 2025 p.12 [text] Placements grew strongly over three years (see Figure 3).
Annual Report 2025 p.12 [figure] images/ar25_p12_fig3.png
Annual Report 2025 p.12 [caption] Figure 3: Students placed per year
Annual Report 2025 p.12 [description] Bar chart: 412 (2023), 471 (2024), 538 (2025). [generated]
| Part of the code | What it does |
|---|---|
related_to column |
Points a caption, description or paragraph at the figure it belongs to. |
| First query | Starts from the search hit and looks up which figure it is related to. |
WHERE e.el_id = ? OR e.related_to = ? |
Fetches the figure itself and every element that points to it. |
JOIN pages ... JOIN documents |
Adds the page number and document title, so each piece can be cited. |
One search hit, on the chart's description, brought back the figure image, its caption and the paragraph that discusses it, all labelled with the document and page for a citation.
Design decisions in practice
A final design principle: every element must be able to cite itself. Whatever its modality, an element should carry enough metadata (document, page, position, time range for audio and video) to tell the user exactly where an answer came from, and to show them the original.