Vision-Language Document Parsing
Parse visually rich pages with vision-language models, keeping figures, tables and positions, and verify generated descriptions.
Reading a page the way a person does
OCR turns pictures of letters into text. It does not understand what it is reading: a chart's bars, a table's structure, an arrow in a flowchart, or which caption belongs to which figure. Layout-aware parsers go further by detecting regions such as tables and figures, but a figure is still just a box labelled "picture".
Vision-language models (VLMs) can look at an image and write about it. Given a picture of a page, a VLM can transcribe its text, rebuild its tables as Markdown, describe its charts ("placements rose from 412 in 2023 to 538 in 2025"), and explain a flowchart step by step. Many recent general-purpose language models accept images as input, and there are also smaller models trained specifically for reading documents. Using them during ingestion is called vision-language parsing.
OCR pipelines versus vision-language parsing
OCR pipelines are precise and cheap for text. Vision-language models understand visuals but must be checked.
Neither approach is best for everything. A common design routes each page or region to the right tool: plain text pages go through a fast parser or OCR; pages with charts, diagrams or complex tables go to a vision-language model; and the outputs are merged into one structured document.
Tools such as Docling (IBM) can run picture classification and generate descriptions of figures with a vision model as part of parsing. MinerU, an open-source project from OpenDataLab, is designed to convert complex PDFs such as scientific papers, with formulas, tables and figures, into Markdown and JSON.
What to extract from each page
A good vision-language parsing step produces, for every page:
- the text, in reading order, with headings marked;
- each table as Markdown (or rows of data);
- each figure with its own caption and a generated description;
- the page number and the position (bounding box) of each element on the page, so an answer can point back to exactly where the information came from;
- a link to the page image, so it can be shown to the user.
Compare the outputs for one annual report page below.
Trust, but verify
Vision-language models can misread a value, swap two bars, or describe a trend that is not there, much as OCR misreads characters. Three habits limit the damage:
- Label generated text as generated. Store descriptions in a separate field (for example
figure_description, withgenerated: true), so answers can say "according to the chart description" and reviewers know what to check. - Keep the original image and link to it, so a user can look at the chart itself.
- Check numbers on a sample. Compare generated values with the source for a few dozen figures before trusting a whole collection.
The program below builds page-level records of the kind a vision-language parsing step would output, keeping text, tables and figures separate and marking what was generated.
Expected output:
p.12 heading Placements
p.12 paragraph The institute continued to strengthen industry partnerships.
p.12 figure (generated) Bar chart: 412 (2023), 471 (2024), 538 (2025).
Every element on the page is a small dictionary with a type and a bounding box (bbox, its position on the page). el.get("text") or el["description"] picks whichever content the element has: headings and paragraphs carry text, while a figure carries a description written by the model. The generated flag is what lets the output mark that description, so nobody mistakes it for words printed in the report.