Keeping the Index Fresh
Keep an index fresh with incremental updates, version labels and safe full rebuilds.
Knowledge keeps changing
On 1 June the institute revises its medical leave rules from 15 days to 10. The new PDF is uploaded to the portal that afternoon. If the knowledge base is not updated, the assistant keeps telling students they have 15 days, and it does so confidently, with a citation, because the old chunk is still in the index. Nothing looks broken. The answer is simply wrong.
A knowledge base is only as current as its index. Freshness means how closely the index matches the documents as they are right now. Keeping it fresh means handling four kinds of change: new documents, edited documents, deleted documents, and changes in who is allowed to see them.
Full rebuilds versus incremental updates
The simplest way to stay fresh is a full rebuild: every night, re-parse, re-chunk and re-embed everything. It is easy to get right, but it gets slower and more expensive as the collection grows, and between rebuilds the index can be up to a day out of date.
Incremental indexing processes only what changed. A fingerprint (hash) of each document's content is stored; on each run, documents whose fingerprint is unchanged are skipped, new ones are added, changed ones are re-processed, and deleted ones are removed.
Only changed documents are processed. Old chunks of edited documents are removed so they cannot be retrieved again.
One detail causes many real bugs: when a document is edited, its old chunks must be removed, not just new chunks added. Otherwise the index holds both versions, and the old 15-day rule can still be retrieved. The safe pattern is to delete every chunk with that document's ID, then insert the new chunks, ideally as one step so that a search in between never sees a half-updated document.
The program below runs an incremental update over a small index.
Expected output:
UPDATE leave (delete old chunks, insert new ones)
SKIP hostel
ADD exam-dates
DELETE circular-14 (remove all its chunks)
fingerprint turns a document's text into a short code that changes whenever the text does. The first loop compares each current document with what the index remembers: an unknown ID is new, a different fingerprint means the text was edited, and the same fingerprint means nothing needs doing. The second loop finds IDs the index still holds that no longer exist in the sources.
Only two of four documents needed processing, and the deleted circular was removed instead of lingering in search results.
Versions and effective dates
Sometimes old versions should be kept rather than deleted. A student who joined in 2024 may still be governed by the 2024 regulations, and staff may need to ask "what was the rule last year?". In that case, keep both versions but label them: a version number, an effective_from date, and a status of current or superseded. Searches for ordinary questions filter to status = current; questions about history can include older versions explicitly. The important thing is that the choice is deliberate, not accidental.
When everything must be re-embedded
Some changes affect every chunk at once:
- switching to a different embedding model, or a new version of the same model;
- changing the chunk size or chunking method;
- fixing a parser bug that affected many documents.
Embeddings from different models cannot be mixed, so a model change means re-embedding the whole collection. For large collections, the safe way is to build the new index alongside the old one, test it, and switch searches over only when it is complete. This is sometimes called a blue-green rebuild. Lineage metadata (which model, chunker and parser produced each chunk) tells you exactly what needs to be redone.
Decide what each change requires below.
How fresh is fresh enough?
Different collections need different freshness. Exam notices may need to appear within minutes; regulations within a day; archived circulars never change. Track two simple numbers: the lag (time between a document changing and the index reflecting it) and the share of indexed documents whose source has changed since they were processed. If either grows, freshness is slipping.