recallA retrieval
field guide

Metadata Engineering

Build this skill

Design and validate a metadata schema that supports citations, filters, permissions and lineage.

Information about the information

Every chunk in a knowledge base carries text and an embedding. It should also carry metadata: information about the chunk, such as which document it came from, which section, when it was last updated, which department owns it, what language it is in and who is allowed to see it.

Metadata is easy to skip when building a prototype, and painful to add later. It is what lets a system:

  • cite sources: "Academic Regulations 2026, Section 4, page 11";
  • filter: search only hostel documents, only current versions, only English, only what this user may see;
  • sort and break ties: prefer the newer of two matching circulars;
  • trace problems: find every chunk that came from a document parsed with a buggy parser version.

Without metadata, a retrieval system can only say "here is some text that looked similar". With it, the system knows where everything came from and whether it should be used.

Designing a metadata schema

A schema is the agreed list of fields every chunk carries, with their meanings and allowed values. A practical schema for a campus knowledge base might look like this:

Field Example Used for
doc_id acad-reg-2026 Linking chunks to their document
title Academic Regulations 2026 Showing citations
section 4. Medical Leave Citations; adding context to chunks
page 11 Linking to the exact page
source_url https://portal.kit.example/… Opening the original
department Academic office Filtering; finding the owner
doc_type policy · circular · FAQ · form Filtering by kind
language en · hi · te Filtering; reporting by language
access students · staff · restricted Permission filtering
effective_from 2026-06-01 Choosing the current version
version 3 Spotting outdated copies
status current · superseded · draft Hiding old and draft versions
tags leave, attendance Simple topic filters

Two rules make a schema useful. Use fixed lists of values wherever possible ("policy", "circular", "FAQ"), not free text, otherwise one team writes "Policy", another "policies" and a third "rules", and filters silently miss documents. And make the important fields required, so that no chunk enters the index without, for example, an access level.

Where metadata comes from

Metadata comes from three places:

  • The source system: connectors read the title, author, folder, dates and permissions.
  • The document itself: the parser finds the section headings, page numbers and language.
  • Rules and models: simple rules ("anything in the /circulars folder is a circular") or a language model asked to suggest tags, a document type or an effective date. Generated metadata should be checked, at least on a sample.
Source systemtitle, dates,accessParsersection, page,languageRules ormodeltype, tags,statusValidaterequiredfields, allowedvaluesChunk +metadatastored together

Metadata is collected from several places, checked against the schema, and stored with every chunk.

Lineage is a special kind of metadata: a record of where a chunk came from and how it was made. It includes the original file, its version, the parser and chunker used and when it was processed. Lineage is what lets you answer "which chunks need re-processing now that the parser has been fixed?" in one query instead of a week of detective work.

Checking metadata automatically

Because filters and permissions depend on metadata, a missing or misspelt field can hide documents or, worse, expose them. The program below validates a few chunks against a small schema.

python · runs in your browser

Expected output:

text
acad-reg-2026  OK
circ-14        doc_type='Circular' not allowed
hr-leave       missing access

The first problems line lists every required field that is absent from a chunk. The second checks each field named in SCHEMA against its set of allowed values. The comparison is exact, so a capital letter is enough to fail.

The circular has "Circular" with a capital C, so a filter for doc_type = circular would silently skip it. The staff leave rules have no access level at all, which is the more dangerous mistake: depending on how the search treats missing values, staff-only rules could be shown to students.

Metadata in action

See how metadata changes the results for the same question below.

Good metadata design pays off far beyond any single feature. It is the foundation for citations, permissions, freshness and debugging, all of which become much harder to bolt on after millions of chunks have been stored without it.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close