recallA retrieval
field guide

Page-image Retrieval

Build this skill

Retrieve page images directly with ColPali-style late interaction and weigh it against text extraction.

Skip parsing: search the page itself

Every pipeline so far has first turned documents into text, with OCR, layout parsers or vision-language models, and then searched the text. Each of those steps can lose something: a table's structure, a chart's values, the layout that shows which numbers belong together. For visually rich documents like annual reports, brochures, slide decks and scientific papers, a lot can be lost.

Page-image retrieval (also called visual document retrieval) takes a different route. It treats each page as a picture, and embeds the picture directly with a model trained to understand document images. Search then finds the page images that best match the question. Nothing has to be parsed before searching, so nothing is lost in parsing.

ColPali

The approach became prominent with ColPali, presented by Manuel Faysse and colleagues in 2024. ColPali combines two ideas:

  • a vision-language model (PaliGemma, from Google) that reads a page image and produces a vector for each small patch of the image;
  • late interaction from ColBERT: the question is turned into one vector per word piece, and each of those finds its best-matching patch on the page; the best-match scores are added up.

So when a student asks "how many students were placed in 2025?", the words "placed" and "2025" each find the patches of the page where the chart's label and the 2025 bar are drawn. The page with the strongest matches ranks first.

Page imagesone per pageVision modelpatch vectorsper pageStoremany vectorsper pageQuestiontoken vectorsLateinteractionbest patch pertoken

Pages are searched as images. Each question token finds its best-matching patch on each page.

The ColPali paper also introduced a benchmark called ViDoRe for retrieval over visually rich documents, on which this approach clearly beat text-extraction pipelines. Related work, such as VDocRAG (CVPR 2025), builds full question-answering systems on top of page-image retrieval.

Step through how one question is answered.

Costs and trade-offs

Page-image retrieval has real strengths: no fragile parsing pipeline, layout and charts preserved, and strong results on visually rich documents. It also has costs.

  • Storage. Each page becomes around a thousand vectors rather than one, so indexes are much larger, though compression and pooling techniques reduce this.
  • Generation needs a vision model. The answer step must read page images, which costs more than reading text.
  • Exact text search is harder. Keyword matching on codes and names needs extracted text anyway.
  • Plain text documents gain little. For typed policies with no figures, text extraction is cheap and accurate.

A practical hybrid

Many teams combine the two worlds: extract text from every page for keyword search and for documents that are mostly text, and add page-image retrieval for visually rich collections. Both point to the same page IDs, so results can be merged, and the answer can cite "Annual Report 2025, page 12" whichever route found it.

The program below shows how page-level results from a text index and an image index can be merged by page ID.

python · runs in your browser

Expected output:

text
annual-report p.13   0.0325
annual-report p.12   0.0164
annual-report p.3    0.0161

The two indexes score on different scales (0.71 against 18.4), so the scores themselves are never compared. Each page earns 1 / (60 + rank) from every list it appears in, and scores.get(page, 0) lets a page found by both routes collect points twice.

Page 13 was found by both routes, so it ranks first; page 12, which only the image index found, still makes the list, where a text-only system would have missed it entirely.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close