How Search Systems Evolved
Describe how search moved from directories to AI answers and trace a query through the indexing and query pipelines.
Finding the one page you need
Meera, a student at the Kaveri Institute of Technology, has been ill for six days. She wants to know how many days of medical leave she is allowed. The answer exists somewhere in the institute's 300 PDFs, circulars and web pages. Her problem is not missing information. Her problem is finding the right paragraph quickly.
That problem is called information retrieval: finding the material in a large collection that answers someone's need. Web search engines are the best-known example, but the same idea powers the search box in your email, on a shopping site and on a company intranet. Every AI assistant that answers questions from documents is, at its heart, a retrieval system with a language model attached.
Three words come up again and again. The information need is what the person really wants to know. The query is what they type, usually much shorter, like "medical leave days". A page is relevant if it actually helps with the need, not just because it contains the query words.
The examples use a small set of 13 pages from the (fictional) Kaveri Institute's student portal: medical leave, hostel fees, library rules and so on. Using one familiar collection makes it easy to compare different ways of searching it.
Four ways people have searched
Search has changed several times, and each change came from the same pressure: there were more pages than people could look through.
| Approach | How you ask | What you get back |
|---|---|---|
| Directory | Click through folders an editor made | Pages filed under that folder |
| Keyword search | Type words | Every page containing those words |
| Ranked search | Type words | Pages sorted, best match first |
| AI answer | Ask a question | A written answer with its sources |
In the mid-1990s, sites like Yahoo were directories: people sorted websites into categories by hand. Quality was good, but editors could not keep up once the web grew to millions of pages, and visitors had to guess the right folder.
Keyword search engines let a program read every page and record every word, so you could just type. The new problem was too many results, in no useful order.
Ranked search fixed that by giving each page a score and showing the best first. Google, launched in 1998, became famous for getting the top ten results right.
The latest step is the AI answer. The system still searches first, then a language model reads the top pages and writes a short answer that cites them.
Each step moved work from the person searching to the system. The AI answer still depends on a search underneath.
Try all four on the same campus pages below. Pick a question, then switch between the four methods and notice who does the work in each one: you, an editor, the engine or a language model.
Notice one thing that never changes. The AI answer is only as good as the pages the search hands it. If the search misses the right page, the answer will be wrong, however well it is written. That is why building AI systems over documents starts with search.
What happens inside a search engine
Almost every search system, from Google to a college portal, has two halves.
The first half runs ahead of time, whenever documents are added or changed. It collects the documents, pulls the text out of each file, splits the text into words and stores those words in an index, a lookup table that makes searching fast.
The second half runs every time someone searches. It cleans up the query the same way, finds the pages that match in the index, ranks them, and shows the results or an answer.
Slow work is done once, ahead of time, so that each search can be fast.
The split matters because the slow work, like reading a 40-page PDF, happens once. Each search then only does the quick part. Whenever you meet a new search system, a useful first question is: which steps happen ahead of time, and which happen during the search? Sort the steps below to practise.
Why searching inside an organisation is hard
A college portal or company intranet has the same problem as the web, plus a few extra difficulties. Web search engines get strong hints from billions of links between pages and from millions of users clicking results. An internal collection has almost no links and far fewer users. Its files are awkward: scanned circulars, spreadsheets, slides saved as PDFs. Old and new versions of the same rule often sit side by side. Some documents are private, so each person may only see some of them.
This is why people often say their company search "never finds anything", even though every document is in there. Much of the work of building a good retrieval system goes into handling exactly these problems.
The simplest search engine
The simplest possible search engine reads every page for every query and counts how many query words it finds. The code below does exactly that for five campus pages. Run it.
Expected output:
[('D3', 1), ('D4', 2)]
[('D2', 1), ('D3', 1), ('D4', 1), ('D6', 1)]
search splits the query into words and, for each page, counts how many of them appear in the page's text. The test word in text.lower() looks for the letters anywhere in the text, not for a whole word. A page with at least one hit goes into the results with its count.
The first query works. The second shows the weaknesses. "late" matched D2 because the letters l-a-t-e sit inside the word "calculated". Every match got the same score of 1, so the engine cannot say which page is best. And it reads every page for every search, which is fine for five pages and far too slow for five million. Real search engines fix all three problems: they split text into proper words, build an index, and give each match a score.