Document Connectors
Describe what a connector does, design incremental syncs that handle deletions, and carry permissions with documents.
Getting the documents in the first place
Before a document can be parsed, cleaned and searched, someone has to fetch it. That sounds trivial when the files sit in one folder on your laptop. In a real organisation they sit in many places at once: a SharePoint site for the academic office, a Google Drive for the placement cell, a Confluence wiki for the IT team, the college website, and email. Each place has its own way of listing files, its own login, its own permissions and its own pace of change.
A connector is the piece of code that reads documents from one source and hands them to the rest of the pipeline in a standard form. A retrieval system usually has several connectors, one per source, all feeding the same parsing and cleaning steps.
What every connector has to do
Whatever the source, a connector does the same five jobs.
Every connector logs in, lists and fetches documents, collects their details, and hands them over in the same format as every other connector.
Authenticate
Log in to the source with a service account or an access token, never with someone's personal password, and with read-only permission wherever possible.
List
Find which documents exist, usually through the source's API (its official interface for programs).
Fetch
Download each document's content.
Read the details
Collect metadata: title, author, folder, last-modified date, web link, and, very importantly, who is allowed to see the document.
Hand over
Pass everything to the next step in one standard shape, so the parser does not care whether a file came from SharePoint or Google Drive.
That standard shape can be very simple. The program below shows a common pattern: each connector produces records with the same fields, so the rest of the pipeline treats them all alike.
Expected output:
google-drive Placement Policy 2026 modified 2026-07-02 visible to ['students', 'staff']
website Hostel Timings modified 2026-06-15 visible to ['everyone']
A dataclass is a class that only holds named fields. from_drive and from_website are adapters: each takes whatever its own source returns and fills in the same seven fields. Everything after the connector reads d.title, d.modified and d.allowed without needing to know where the document came from.
Keeping up with changes
Fetching everything once is the easy part. Documents change constantly: new circulars arrive, policies are revised, old files are deleted. Re-downloading the whole collection every night wastes time and money, and can hit the source's rate limits (caps on how many requests you may make).
Most connectors therefore do an incremental sync: they only fetch what changed since the last run. There are two common ways to know what changed. Polling means asking the source, on a schedule, "what changed since this time yesterday?" using last-modified dates. Change notifications (sometimes called webhooks) mean the source tells your system as soon as something changes, which is faster but more complex to set up.
Deletions need special care. If a document is deleted or someone's access is removed in the source, the retrieval system must remove it too. Otherwise the assistant keeps quoting a withdrawn circular, or keeps showing a confidential file to people who lost access. This is one of the most commonly forgotten parts of a connector.
Step through one nightly sync below.
Reading permissions, not just content
Every serious source has its own permission system: who can view each file, folder or page. A retrieval system that ignores this becomes a way around it. If a confidential staff circular is indexed with no record of who may see it, any student's question could pull it into an answer.
So connectors should read each document's access list along with its content, store it as metadata on every chunk, and update it when it changes. The search step can then filter results by the person asking. Getting permissions right is mostly a connector job, because only the connector can see the source's rules.
Decide which approach fits each source below.