recallA retrieval
field guide

Building an Ingestion Pipeline

Build this skill

Assemble an idempotent ingestion pipeline that handles failures, records what happened and can be measured.

Putting the pieces in one line

An ingestion pipeline is the chain of steps that takes documents from where they live and turns them into clean, structured, labelled text ready to be split into chunks. Each step has its own job, and the output of one is the input of the next:

  1. Fetch documents from each source with a connector.
  2. Parse each file into structured text, using OCR for scans.
  3. Clean the text: remove duplicates and boilerplate, repair and normalise.
  4. Protect sensitive data: detect it, then drop, mask or label it.
  5. Enrich with metadata: source, title, section, date, language and access.
  6. Store the result, ready for chunking and embedding.
FetchconnectorsParseto Markdown,OCRCleandedupe,normaliseProtectPII, labelsEnrichmetadataStoreready tochunk

Each document flows through the same steps, and each step's output is saved so problems can be traced.

Building each step well is only half the job. The other half is making the whole chain dependable: it must handle thousands of documents, survive failures, avoid doing the same work twice, and make it easy to find out what happened to any single file.

A pipeline you can run

The program below is a tiny but complete pipeline. Each step is a small function, and the pipeline passes each document through them in order. The parsing step is a stand-in, but the structure is the same one you would use with real tools.

python · runs in your browser

Expected output:

text
skip duplicate: leave-copy.pdf
hostel.html 5c13aeaca43d  Hostel gates close at 10 PM. Warden: [PHONE]
leave.pdf  895797697be2  Students may take up to 10 days of medical leave.
Step What it does
clean Drops the menu line, then collapses every run of spaces and line breaks into one space.
protect Replaces any ten-digit mobile number with [PHONE].
enrich Adds a hash, a short fingerprint of the cleaned text, and a word count.
if doc["hash"] in seen Two documents with the same fingerprint have the same text, so the second is skipped.

Three documents went in and two came out: the menu was removed, the phone number masked, and the emailed copy of the leave rule recognised as a duplicate by its fingerprint.

Design rules that make pipelines dependable

Make every step idempotent

An idempotent step gives the same result no matter how many times it runs on the same input. If the pipeline crashes halfway through and is restarted, nothing should be duplicated or lost. Fingerprints help here: if a document's content hash has not changed, the pipeline can skip it.

Save the output of each step

Keep the raw file, the parsed Markdown and the cleaned text separately. When an answer turns out to be wrong, you can look at each stage and see exactly where the problem started. It also means improving one step, such as switching to a better parser, does not require fetching everything again.

Expect failures, and keep going

Some PDFs are corrupt, some are password-protected, some are 900 pages long. One bad file should not stop the other ten thousand. Record the failure with its reason, move on, and review failures later.

Record what happened

For each document, keep a short log: when it was fetched, which parser and version processed it, how long it took, what was masked and whether it succeeded. This is how you answer questions like "which documents still use the old parser?" or "why is this circular missing?".

Walk through how the pipeline handles four different files below.

Measuring the pipeline

Once a pipeline runs, a few numbers tell you whether it is healthy:

Measure What it tells you
Documents processed per hour Whether a full rebuild finishes in time
Failure rate, by reason Which file types or sources cause trouble
Time per step Where the bottleneck is (usually parsing or OCR)
Share of documents flagged for review How much human checking is needed
Duplicates removed How messy the sources are

Use the calculator below to see how long a full run takes and where the time goes.

Even a modest share of scanned documents usually dominates the total time, which is why OCR is the first place to look when a pipeline is too slow, and why incremental runs that skip unchanged files matter so much.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close