Handling Sensitive Data at Ingestion
Detect personal data at ingestion and decide whether to drop, mask or restrict it, with labels that travel with the data.
Some documents contain more than rules
A college's shared drive does not only hold regulations and circulars. It also holds leave applications with students' names and phone numbers, scholarship forms with family income, fee receipts with bank details, and medical certificates. A company drive holds salary sheets, customer records and contracts. If all of this goes into a retrieval system without care, a student asking "what is the medical leave policy?" could receive a passage from another student's medical certificate.
Information that can identify a person is called personally identifiable information, or PII: names combined with other details, phone numbers, email addresses, roll numbers, Aadhaar and PAN numbers, bank account numbers, addresses, and health information. Handling it safely is much easier at the point where documents enter the system, before they are chunked, embedded and copied into indexes, caches and logs.
Why ingestion is the right place
Once a passage has been embedded and stored, the sensitive text now exists in several places: the stored chunk, its embedding, backups, and possibly logs of every prompt that used it. Removing it later means finding every copy. Dealing with it at ingestion keeps it out of all of them at once.
Laws point the same way. India's Digital Personal Data Protection Act, 2023 (DPDP Act) requires organisations to collect and use personal data only for a clear, lawful purpose and to protect it with reasonable safeguards. Feeding every scanned form into a general-purpose chatbot is hard to justify under that principle. Similar rules exist elsewhere, such as the GDPR in Europe.
Detect sensitive data, decide what to do with it, label the document, and only then index it.
Detecting sensitive data
Detection uses two kinds of tools.
Patterns catch data with a fixed format. An Indian mobile number is ten digits, usually starting with 6, 7, 8 or 9. A PAN has five letters, four digits and a letter. An email address has an @ and a domain. These can be matched with regular expressions, short patterns that describe text, and they are fast and predictable.
Named-entity recognition catches data without a fixed format, such as people's names, addresses and organisations. A trained model reads the text and labels words like "Meera Reddy" as a person. Open-source tools such as Microsoft Presidio combine both approaches and let you add your own patterns.
The program below uses patterns to find and mask phone numbers, email addresses and PAN numbers in a leave application.
Expected output:
Leave application from Meera Reddy, roll no [ROLL]. Contact: [PHONE], [EMAIL]. PAN of parent: [PAN]. Requesting 6 days of medical leave from 3 March.
| Pattern | What it matches |
|---|---|
\b[6-9]\d{9}\b |
A ten-digit number starting with 6, 7, 8 or 9: an Indian mobile number. \b marks a word boundary, so longer numbers are not matched in part. |
[\w.]+@[\w.]+\.\w+ |
Letters, digits or dots, then @, then a domain with a dot in it: an email address. |
[A-Z]{5}\d{4}[A-Z] |
Five capital letters, four digits and one capital letter: the shape of a PAN. |
re.sub replaces every match of a pattern with its label, and the loop applies the four patterns one after another. They caught the phone number, email, PAN and roll number, but not the name "Meera Reddy", because names do not follow a pattern. That is the job of a named-entity model, and it is why real systems use both.
Patterns can also be too greedy. In the lab, mask what should be masked without destroying the reason the document exists.
Drop, mask or keep?
Detecting sensitive data is only half the decision. The other half is what to do with it, and that depends on the purpose of the system.
- Drop the document when it has no place in the system at all. A campus helpdesk has no reason to index individual leave applications.
- Mask or redact the sensitive parts when the rest is useful, replacing "9876543210" with "[PHONE]".
- Keep it, with restricted access, when authorised people genuinely need it, for example a staff assistant that helps the exam cell look up a specific student's re-evaluation request. The document is then labelled as sensitive, and only searchable by people with the right role.
Practise the decision below.
Labels travel with the data
Whatever is kept should carry a sensitivity label, such as public, internal, confidential or restricted, stored as metadata with every chunk made from the document. Later, at search time, the system can use that label together with the user's role to decide what each person is allowed to retrieve. A label added at ingestion is cheap; reconstructing which chunks came from sensitive documents after the fact is not.
Two more habits help. Keep a record of what was detected and what was done, so you can answer the question "did any Aadhaar numbers get indexed?" with evidence. And remember that masking only protects the text you index: the original files, logs and backups need their own protection.