Metadata Engineering
Design and validate a metadata schema that supports citations, filters, permissions and lineage.
Information about the information
Every chunk in a knowledge base carries text and an embedding. It should also carry metadata: information about the chunk, such as which document it came from, which section, when it was last updated, which department owns it, what language it is in and who is allowed to see it.
Metadata is easy to skip when building a prototype, and painful to add later. It is what lets a system:
- cite sources: "Academic Regulations 2026, Section 4, page 11";
- filter: search only hostel documents, only current versions, only English, only what this user may see;
- sort and break ties: prefer the newer of two matching circulars;
- trace problems: find every chunk that came from a document parsed with a buggy parser version.
Without metadata, a retrieval system can only say "here is some text that looked similar". With it, the system knows where everything came from and whether it should be used.
Designing a metadata schema
A schema is the agreed list of fields every chunk carries, with their meanings and allowed values. A practical schema for a campus knowledge base might look like this:
| Field | Example | Used for |
|---|---|---|
| doc_id | acad-reg-2026 |
Linking chunks to their document |
| title | Academic Regulations 2026 | Showing citations |
| section | 4. Medical Leave | Citations; adding context to chunks |
| page | 11 | Linking to the exact page |
| source_url | https://portal.kit.example/… | Opening the original |
| department | Academic office | Filtering; finding the owner |
| doc_type | policy · circular · FAQ · form | Filtering by kind |
| language | en · hi · te | Filtering; reporting by language |
| access | students · staff · restricted | Permission filtering |
| effective_from | 2026-06-01 | Choosing the current version |
| version | 3 | Spotting outdated copies |
| status | current · superseded · draft | Hiding old and draft versions |
| tags | leave, attendance | Simple topic filters |
Two rules make a schema useful. Use fixed lists of values wherever possible ("policy", "circular", "FAQ"), not free text, otherwise one team writes "Policy", another "policies" and a third "rules", and filters silently miss documents. And make the important fields required, so that no chunk enters the index without, for example, an access level.
Where metadata comes from
Metadata comes from three places:
- The source system: connectors read the title, author, folder, dates and permissions.
- The document itself: the parser finds the section headings, page numbers and language.
- Rules and models: simple rules ("anything in the /circulars folder is a circular") or a language model asked to suggest tags, a document type or an effective date. Generated metadata should be checked, at least on a sample.
Metadata is collected from several places, checked against the schema, and stored with every chunk.
Lineage is a special kind of metadata: a record of where a chunk came from and how it was made. It includes the original file, its version, the parser and chunker used and when it was processed. Lineage is what lets you answer "which chunks need re-processing now that the parser has been fixed?" in one query instead of a week of detective work.
Checking metadata automatically
Because filters and permissions depend on metadata, a missing or misspelt field can hide documents or, worse, expose them. The program below validates a few chunks against a small schema.
Expected output:
acad-reg-2026 OK
circ-14 doc_type='Circular' not allowed
hr-leave missing access
The first problems line lists every required field that is absent from a chunk. The second checks each field named in SCHEMA against its set of allowed values. The comparison is exact, so a capital letter is enough to fail.
The circular has "Circular" with a capital C, so a filter for doc_type = circular would silently skip it. The staff leave rules have no access level at all, which is the more dangerous mistake: depending on how the search treats missing values, staff-only rules could be shown to students.
Metadata in action
See how metadata changes the results for the same question below.
Good metadata design pays off far beyond any single feature. It is the foundation for citations, permissions, freshness and debugging, all of which become much harder to bolt on after millions of chunks have been stored without it.