Cleaning and Normalizing Text
Remove exact and near-duplicates, strip boilerplate and normalise text without cleaning away meaning.
Parsed text is not yet clean text
After parsing, you have text, but rarely clean text. The same circular appears three times because it was emailed, uploaded to the portal and saved on the shared drive. Every web page starts with the same navigation menu. Scanned pages carry stray characters from stamps. Some text uses curly quotes and some straight ones; some has invisible characters copied from web pages. None of this is dramatic on its own, but together it makes search results noisy and wastes the limited space in a language model's prompt.
Cleaning removes what should not be there. Normalising makes things that mean the same look the same. Both happen before text is split into chunks and embedded.
Removing duplicates
Duplicates are the most common problem in real collections, and they cause a specific kind of damage. If the same passage is stored five times, a search returns it five times, filling the top results with copies and pushing other useful passages out.
Exact duplicates are easy to catch. Compute a hash of each document's text, a short fingerprint that is identical for identical text, and keep only one document per fingerprint.
Near-duplicates are harder and more common: the 2025 and 2026 versions of the same policy, a circular with a different date at the top, a web page and its printer-friendly copy. A standard technique is to compare shingles, short overlapping runs of words. "Students may take up to 10 days" becomes the shingles "students may take", "may take up", "take up to" and so on. Two documents that share most of their shingles are near-duplicates. The overlap is measured with Jaccard similarity: shared shingles divided by all distinct shingles. For millions of documents, a faster approximation called MinHash estimates the same number without comparing every pair.
Expected output:
2025 vs 2026 policy: 0.50
policy vs fee page: 0.00
shingles slides a three-word window along the text and collects every window in a set. In jaccard, sa & sb is the set of shingles the two texts share and sa | sb is every shingle found in either, so the result is the shared fraction.
The two versions of the policy share half of their shingles, even though they differ only by one number, because every shingle that touches the changed number is different. The fee page shares none. In practice, a score well above that of unrelated pages (often 0.5 or more for short texts) is a strong hint of a near-duplicate worth checking. Notice what this means in practice: near-duplicate detection tells you the two policies are almost the same, but which one to keep is a different decision. Here the newer version should win, and the older one should be removed or marked as superseded, because the one changed number is the most important fact in it.
Removing boilerplate
Boilerplate is text that repeats across many documents but says nothing about any of them: website menus, cookie notices, "Page 3 of 42", letterheads, email signatures and legal disclaimers. It pollutes search in two ways. It adds noise to every passage's embedding, and it can make unrelated documents look similar because they share the same footer.
A simple, effective trick is to count how often each line appears across the collection. A line that appears on hundreds of pages, such as "Home | About | Admissions | Contact", is almost certainly boilerplate.
Repairing and normalising text
Several smaller fixes make text consistent:
| Problem | Example | Fix |
|---|---|---|
| Broken line endings | "medi-\ncal leave" | Join hyphenated words split across lines |
| Extra whitespace | "hostel fee" | Collapse repeated spaces |
| Curly versus straight quotes | “leave” vs "leave" | Pick one form |
| Invisible characters | zero-width spaces from web pages | Remove them |
| Different Unicode forms | the same Hindi or accented letter stored two ways | Apply Unicode normalisation |
The last row needs a short explanation. Unicode is the standard that gives every character in every script a number. Some characters, especially in Indian scripts and accented letters, can be stored in more than one way that looks identical on screen. Unicode normalisation (often the forms called NFC or NFKC) converts them to a single standard form, so that a search for one matches the other.
Each step is simple. Together they make search results cleaner and save prompt space.
Don't clean away meaning
Cleaning can go too far. Lowercasing everything loses the difference between "CS" (computer science) and "cs". Removing all numbers destroys fees, dates and roll numbers, which are the facts people ask about. Stripping every symbol breaks course codes, email addresses and formulas. Unlike the analysis done inside a keyword index, cleaning changes the text that the language model will read and quote, so it should stay readable and faithful to the original.
Try it on a page from the handbook. Five of the steps repair the text. Two of them go too far.
A good rule is to clean conservatively, keep a link to the original file, and check samples before and after.