Enterprise Documents and Their Formats
Classify documents as structured, semi-structured or unstructured, tell born-digital from scanned, and profile a collection.
Knowledge is scattered across many kinds of file
Ask where the Kaveri Institute keeps its rules and the honest answer is "everywhere". The academic regulations are a 42-page PDF. The fee structure is an Excel sheet. Circulars are scanned letters, signed and stamped. The hostel handbook is a Word file, the placement policy lives on a web page, and the exam timetable is a slide deck someone exported to PDF. A company is the same, only bigger: contracts, wikis, spreadsheets, emails, slide decks and support tickets.
Before any of this can be searched, it must be turned into clean text. This step is easy to underestimate. A retrieval system can only find what it was able to read, and every mistake made while reading a document, such as a lost table, a garbled scan or a missing heading, is carried silently into every answer built on it. Understanding the kinds of documents you are dealing with is the first step to reading them well.
Structured, semi-structured and unstructured
Documents are often grouped by how much structure they have.
| Kind | What it looks like | Examples | How hard to read |
|---|---|---|---|
| Structured | Rows and columns with fixed fields | Fee database, Excel sheet of marks, attendance records | Easy to read, but often better queried than searched |
| Semi-structured | Text with some markup or fields | Web pages (HTML), JSON, emails, Markdown, forms | Medium: the structure helps if you keep it |
| Unstructured | Free-flowing text and layout | PDFs, Word files, slides, scanned letters | Hardest: structure has to be recovered |
Structured data lives in tables with named columns, so a question like "what is the hostel fee?" can often be answered by looking up a cell rather than by searching text. Semi-structured data, such as an HTML page, has tags marking headings, lists and links, which are very useful clues if your reader keeps them. Unstructured documents, such as PDFs and scans, are mostly text and layout, and their structure (which line is a heading, which numbers form a table) has to be worked out.
Two PDFs can look identical on screen. One contains real text; the other is only a picture of text.
Born-digital or scanned?
PDFs deserve special attention because they are everywhere and they come in two very different kinds. A born-digital PDF was created by software, for example by exporting a Word file. It contains the actual characters, so text can be copied out directly. A scanned PDF is a photograph of a paper page. It contains no characters at all, just pixels, and the text must be recognised by OCR (optical character recognition) before it can be searched.
You cannot tell which kind you have by looking. A quick test is to try selecting the text: if you can highlight individual words, it is probably born-digital. Many collections contain both, and some single PDFs contain both, such as a typed report with a signed, scanned page at the end.
Why each format causes its own problems
Every common format has a typical way of going wrong.
- PDF: stores characters at positions on a page, not paragraphs in order. Two-column layouts, headers, footers and page numbers can end up mixed into the text.
- Word (DOCX): usually reads well, but tracked changes, comments and text boxes can appear in odd places.
- Slides (PPTX): short bullet fragments with little context; much of the meaning is in pictures and the speaker's head.
- Spreadsheets (XLSX): tables without sentences; a cell like "96000" means nothing without its row and column headers.
- Web pages (HTML): menus, cookie banners and footers surround the real content.
- Scans: recognition errors, stamps, signatures, handwriting and tilted pages.
Sort the problems below by format.
Profiling a collection before you start
Before building anything, it pays to take an inventory of the collection: how many files of each type, how many are scanned, how many have tables, and how big they are. This tells you where the effort will go. A collection that is 90 percent born-digital Word files needs a simple pipeline; one that is half scanned circulars needs good OCR and a lot of checking.
The short program below profiles a small list of files and counts them by format and difficulty. In a real project, the same idea runs over a whole shared drive.
Expected output:
by format: {'pdf': 3, 'xlsx': 1, 'docx': 1, 'html': 1, 'pptx': 1}
by kind: {'born-digital': 2, 'structured': 1, 'scanned': 2, 'web': 1, 'slides': 1}
needs OCR: 2 of 7 -> ['Circular_14_Leave.pdf', 'Circular_22_Holidays.pdf']
Counter tallies how many times each value appears. f[1] is a file's format and f[2] its kind, so the two Counter lines produce the two summaries, and the list comprehension picks out the names of the scanned files.
Two of the seven files are scans, and one is a spreadsheet that might be better queried as data than searched as text. Those numbers shape the plan before a single line of the pipeline is written.
Where knowledge actually lives
One more complication: documents are rarely in one place. A college might keep regulations on its website, circulars in email, fee data in an accounting system and timetables on a shared drive. A company spreads knowledge across SharePoint, Google Drive, Confluence, ticketing systems and chat. Each source has its own way of being accessed, its own permissions and its own pace of change. Part of designing a document pipeline is deciding which sources matter, who owns them, and how often they need to be read again.