Parsing Documents into Markdown
Explain why plain text extraction fails on real PDFs and convert documents to structured Markdown with layout-aware parsers.
From file to clean, structured text
Parsing means reading a file and extracting its content in a form a program can use. For retrieval, the goal is not just the words but the words with their structure: which lines are headings, which items form a list, which numbers belong to which table cell, and in what order a person would read them. Losing that structure makes later steps much harder. A chunk cut from the middle of a policy without its heading may not say what it is about; a table flattened into a row of numbers may be impossible to interpret.
A popular target format is Markdown, a simple way of writing structured text with plain characters: # for headings, - for list items and | to draw tables. Markdown keeps the structure that matters, stays readable for people, and is easy for language models to understand, because models have seen enormous amounts of it during training.
What naive extraction gets wrong
The simplest way to get text out of a PDF is to ask a basic library for "all the text". For a simple, single-column document this works. For real documents it often produces a mess, because a PDF stores characters at positions on a page, not paragraphs in order. Compare three ways of reading the same page below.
How layout-aware parsers work
Modern layout-aware parsers look at a page more like a person does. They first detect regions on the page, such as headings, paragraphs, tables, figures, headers and footers, often using a small vision model trained on many document images. Then they work out the reading order across columns, rebuild tables cell by cell, drop repeated headers and footers, and write everything out in a structured format.
The parser finds regions on each page, puts them in reading order and writes them out with their structure.
Two open-source tools are widely used for this.
Docling, from IBM Research, converts PDFs, Word files, slides, HTML and images into a structured document that can be exported as Markdown or JSON. It uses layout-detection and table-structure models, so it handles multi-column pages and tables well, and it can run OCR for scanned pages. It is thorough, which also makes it slower than simple extractors.
MarkItDown, from Microsoft, converts many file types (PDF, Word, PowerPoint, Excel, HTML and more) into Markdown with a focus on speed and simplicity. It works very well on documents that already have clear structure, such as Word and HTML files, and is lighter on difficult PDF layouts.
Using either takes only a few lines. These examples run on your own computer after installing the package, so they are shown as code to copy:
# pip install docling
from docling.document_converter import DocumentConverter
result = DocumentConverter().convert("Academic_Regulations_2026.pdf")
markdown = result.document.export_to_markdown()
# pip install 'markitdown[all]'
from markitdown import MarkItDown
markdown = MarkItDown().convert("Hostel_Handbook.docx").text_content
Choosing a parser
There is no single best parser. A common pattern is to send each file type to the tool that suits it, and to test each choice on real samples from your collection.
| Situation | Reasonable choice |
|---|---|
| Word, HTML, simple PDFs | A fast converter such as MarkItDown |
| Complex PDFs with columns and tables | A layout-aware parser such as Docling |
| Scanned pages | A parser with OCR switched on |
| Huge volumes of simple files | The fastest tool that passes your quality checks |
Checking that parsing worked
Parsing fails quietly. A parser will not raise an error because it merged two columns or dropped a table; it will just produce worse text. So it is worth checking a sample of output by eye, and adding simple automatic checks. The program below runs three such checks on a parsed page: that it has headings, that table rows have a consistent number of cells, and that no footer text has leaked in.
Expected output:
headings found: 1
table rows: 4 | cells per row: {3}
consistent table: True
footer leaks: none
| Line | What it checks |
|---|---|
l.startswith("#") |
Markdown headings begin with #. A page with none has probably lost its structure. |
{l.count("|") - 1 for l in table_rows} |
The number of cells in each table row, collected in a set. One value in the set means every row has the same shape. |
"Page" in l and "of" in l |
A rough test for a footer such as "Page 14 of 96" that has leaked into the text. |
Checks like these will not catch every problem, but they catch the common ones across thousands of files, and they point you to the documents worth reading by hand.