recallA retrieval
field guide

Vision-Language Document Parsing

Build this skill

Parse visually rich pages with vision-language models, keeping figures, tables and positions, and verify generated descriptions.

Reading a page the way a person does

OCR turns pictures of letters into text. It does not understand what it is reading: a chart's bars, a table's structure, an arrow in a flowchart, or which caption belongs to which figure. Layout-aware parsers go further by detecting regions such as tables and figures, but a figure is still just a box labelled "picture".

Vision-language models (VLMs) can look at an image and write about it. Given a picture of a page, a VLM can transcribe its text, rebuild its tables as Markdown, describe its charts ("placements rose from 412 in 2023 to 538 in 2025"), and explain a flowchart step by step. Many recent general-purpose language models accept images as input, and there are also smaller models trained specifically for reading documents. Using them during ingestion is called vision-language parsing.

OCR pipelines versus vision-language parsing

OCR + layout pipelineDetect regions, read charactersFast and cheap per pageExact text when the scan is cleanCharts and diagrams stay boxesTables need special modelsVision-language parsingA model looks at the whole pageSlower and costlier per pageDescribes charts, diagrams, photosRebuilds tables and reading orderCan misread or invent details

OCR pipelines are precise and cheap for text. Vision-language models understand visuals but must be checked.

Neither approach is best for everything. A common design routes each page or region to the right tool: plain text pages go through a fast parser or OCR; pages with charts, diagrams or complex tables go to a vision-language model; and the outputs are merged into one structured document.

Tools such as Docling (IBM) can run picture classification and generate descriptions of figures with a vision model as part of parsing. MinerU, an open-source project from OpenDataLab, is designed to convert complex PDFs such as scientific papers, with formulas, tables and figures, into Markdown and JSON.

What to extract from each page

A good vision-language parsing step produces, for every page:

  • the text, in reading order, with headings marked;
  • each table as Markdown (or rows of data);
  • each figure with its own caption and a generated description;
  • the page number and the position (bounding box) of each element on the page, so an answer can point back to exactly where the information came from;
  • a link to the page image, so it can be shown to the user.

Compare the outputs for one annual report page below.

Trust, but verify

Vision-language models can misread a value, swap two bars, or describe a trend that is not there, much as OCR misreads characters. Three habits limit the damage:

  • Label generated text as generated. Store descriptions in a separate field (for example figure_description, with generated: true), so answers can say "according to the chart description" and reviewers know what to check.
  • Keep the original image and link to it, so a user can look at the chart itself.
  • Check numbers on a sample. Compare generated values with the source for a few dozen figures before trusting a whole collection.

The program below builds page-level records of the kind a vision-language parsing step would output, keeping text, tables and figures separate and marking what was generated.

python · runs in your browser

Expected output:

text
p.12 heading               Placements
p.12 paragraph             The institute continued to strengthen industry partnerships.
p.12 figure    (generated) Bar chart: 412 (2023), 471 (2024), 538 (2025).

Every element on the page is a small dictionary with a type and a bounding box (bbox, its position on the page). el.get("text") or el["description"] picks whichever content the element has: headings and paragraphs carry text, while a figure carries a description written by the model. The generated flag is what lets the output mark that description, so nobody mistakes it for words printed in the report.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close