Building an Ingestion Pipeline
Assemble an idempotent ingestion pipeline that handles failures, records what happened and can be measured.
Putting the pieces in one line
An ingestion pipeline is the chain of steps that takes documents from where they live and turns them into clean, structured, labelled text ready to be split into chunks. Each step has its own job, and the output of one is the input of the next:
- Fetch documents from each source with a connector.
- Parse each file into structured text, using OCR for scans.
- Clean the text: remove duplicates and boilerplate, repair and normalise.
- Protect sensitive data: detect it, then drop, mask or label it.
- Enrich with metadata: source, title, section, date, language and access.
- Store the result, ready for chunking and embedding.
Each document flows through the same steps, and each step's output is saved so problems can be traced.
Building each step well is only half the job. The other half is making the whole chain dependable: it must handle thousands of documents, survive failures, avoid doing the same work twice, and make it easy to find out what happened to any single file.
A pipeline you can run
The program below is a tiny but complete pipeline. Each step is a small function, and the pipeline passes each document through them in order. The parsing step is a stand-in, but the structure is the same one you would use with real tools.
Expected output:
skip duplicate: leave-copy.pdf
hostel.html 5c13aeaca43d Hostel gates close at 10 PM. Warden: [PHONE]
leave.pdf 895797697be2 Students may take up to 10 days of medical leave.
| Step | What it does |
|---|---|
clean |
Drops the menu line, then collapses every run of spaces and line breaks into one space. |
protect |
Replaces any ten-digit mobile number with [PHONE]. |
enrich |
Adds a hash, a short fingerprint of the cleaned text, and a word count. |
if doc["hash"] in seen |
Two documents with the same fingerprint have the same text, so the second is skipped. |
Three documents went in and two came out: the menu was removed, the phone number masked, and the emailed copy of the leave rule recognised as a duplicate by its fingerprint.
Design rules that make pipelines dependable
Make every step idempotent
An idempotent step gives the same result no matter how many times it runs on the same input. If the pipeline crashes halfway through and is restarted, nothing should be duplicated or lost. Fingerprints help here: if a document's content hash has not changed, the pipeline can skip it.
Save the output of each step
Keep the raw file, the parsed Markdown and the cleaned text separately. When an answer turns out to be wrong, you can look at each stage and see exactly where the problem started. It also means improving one step, such as switching to a better parser, does not require fetching everything again.
Expect failures, and keep going
Some PDFs are corrupt, some are password-protected, some are 900 pages long. One bad file should not stop the other ten thousand. Record the failure with its reason, move on, and review failures later.
Record what happened
For each document, keep a short log: when it was fetched, which parser and version processed it, how long it took, what was masked and whether it succeeded. This is how you answer questions like "which documents still use the old parser?" or "why is this circular missing?".
Walk through how the pipeline handles four different files below.
Measuring the pipeline
Once a pipeline runs, a few numbers tell you whether it is healthy:
| Measure | What it tells you |
|---|---|
| Documents processed per hour | Whether a full rebuild finishes in time |
| Failure rate, by reason | Which file types or sources cause trouble |
| Time per step | Where the bottleneck is (usually parsing or OCR) |
| Share of documents flagged for review | How much human checking is needed |
| Duplicates removed | How messy the sources are |
Use the calculator below to see how long a full run takes and where the time goes.
Even a modest share of scanned documents usually dominates the total time, which is why OCR is the first place to look when a pipeline is too slow, and why incremental runs that skip unchanged files matter so much.