recallA retrieval
field guide

Cleaning and Normalizing Text

Build this skill

Remove exact and near-duplicates, strip boilerplate and normalise text without cleaning away meaning.

Parsed text is not yet clean text

After parsing, you have text, but rarely clean text. The same circular appears three times because it was emailed, uploaded to the portal and saved on the shared drive. Every web page starts with the same navigation menu. Scanned pages carry stray characters from stamps. Some text uses curly quotes and some straight ones; some has invisible characters copied from web pages. None of this is dramatic on its own, but together it makes search results noisy and wastes the limited space in a language model's prompt.

Cleaning removes what should not be there. Normalising makes things that mean the same look the same. Both happen before text is split into chunks and embedded.

Removing duplicates

Duplicates are the most common problem in real collections, and they cause a specific kind of damage. If the same passage is stored five times, a search returns it five times, filling the top results with copies and pushing other useful passages out.

Exact duplicates are easy to catch. Compute a hash of each document's text, a short fingerprint that is identical for identical text, and keep only one document per fingerprint.

Near-duplicates are harder and more common: the 2025 and 2026 versions of the same policy, a circular with a different date at the top, a web page and its printer-friendly copy. A standard technique is to compare shingles, short overlapping runs of words. "Students may take up to 10 days" becomes the shingles "students may take", "may take up", "take up to" and so on. Two documents that share most of their shingles are near-duplicates. The overlap is measured with Jaccard similarity: shared shingles divided by all distinct shingles. For millions of documents, a faster approximation called MinHash estimates the same number without comparing every pair.

python · runs in your browser

Expected output:

text
2025 vs 2026 policy: 0.50
policy vs fee page:  0.00

shingles slides a three-word window along the text and collects every window in a set. In jaccard, sa & sb is the set of shingles the two texts share and sa | sb is every shingle found in either, so the result is the shared fraction.

The two versions of the policy share half of their shingles, even though they differ only by one number, because every shingle that touches the changed number is different. The fee page shares none. In practice, a score well above that of unrelated pages (often 0.5 or more for short texts) is a strong hint of a near-duplicate worth checking. Notice what this means in practice: near-duplicate detection tells you the two policies are almost the same, but which one to keep is a different decision. Here the newer version should win, and the older one should be removed or marked as superseded, because the one changed number is the most important fact in it.

Removing boilerplate

Boilerplate is text that repeats across many documents but says nothing about any of them: website menus, cookie notices, "Page 3 of 42", letterheads, email signatures and legal disclaimers. It pollutes search in two ways. It adds noise to every passage's embedding, and it can make unrelated documents look similar because they share the same footer.

A simple, effective trick is to count how often each line appears across the collection. A line that appears on hundreds of pages, such as "Home | About | Admissions | Contact", is almost certainly boilerplate.

Repairing and normalising text

Several smaller fixes make text consistent:

Problem Example Fix
Broken line endings "medi-\ncal leave" Join hyphenated words split across lines
Extra whitespace "hostel fee" Collapse repeated spaces
Curly versus straight quotes “leave” vs "leave" Pick one form
Invisible characters zero-width spaces from web pages Remove them
Different Unicode forms the same Hindi or accented letter stored two ways Apply Unicode normalisation

The last row needs a short explanation. Unicode is the standard that gives every character in every script a number. Some characters, especially in Indian scripts and accented letters, can be stored in more than one way that looks identical on screen. Unicode normalisation (often the forms called NFC or NFKC) converts them to a single standard form, so that a search for one matches the other.

Parsed textraw outputRemoveduplicatesexact and nearStripboilerplatemenus, footersRepairhyphens, OCRjunkNormalisespaces, quotes,Unicode

Each step is simple. Together they make search results cleaner and save prompt space.

Don't clean away meaning

Cleaning can go too far. Lowercasing everything loses the difference between "CS" (computer science) and "cs". Removing all numbers destroys fees, dates and roll numbers, which are the facts people ask about. Stripping every symbol breaks course codes, email addresses and formulas. Unlike the analysis done inside a keyword index, cleaning changes the text that the language model will read and quote, so it should stay readable and faithful to the original.

Try it on a page from the handbook. Five of the steps repair the text. Two of them go too far.

A good rule is to clean conservatively, keep a link to the original file, and check samples before and after.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close