Where Lexical Search Falls Short
Diagnose vocabulary mismatch, polysemy and missing intent in lexical search, and judge when hand-built synonyms help or hurt.
Same question, different words
Three first-year students want to know when the hostel gates close. One types "hostel timings", one types "hostel curfew", and one types "what time do I need to be back in hostel". The handbook page that answers all three says: "Hostel gates close at 10:00 PM for first-year students."
A keyword search engine handles the first query well. For the second, the word "curfew" appears nowhere in the collection, so the engine can only match on "hostel". The timings page ends up third, behind a notice about committee elections and the fee page. The third query does no better.
This is the vocabulary mismatch problem: the person searching and the person who wrote the page use different words for the same thing. It is very common. In a well-known 1987 study, researchers asked people to name everyday objects and computer commands, and found that two people picked the same word less than 20 percent of the time.
Search methods that work by matching words are called lexical search. Everything covered so far, from Boolean queries to BM25, is lexical: a page scores points only for the words it shares with the query. When the words differ, the score is zero, however well the page answers the question.
Synonyms: different words, same meaning
The simplest kind of mismatch is synonyms, different words that mean the same thing in context.
| What the student types | What the page says |
|---|---|
| sick days | medical leave |
| vacation | Diwali break, winter break |
| exam paper rechecked | re-evaluation of answer scripts |
| internet not working | Wi-Fi keeps disconnecting |
| curfew | gates close at 10:00 PM |
Stemming does not help here. It only joins different endings of the same word, like "fees" and "fee". "Sick" and "medical" are different words altogether.
The wider gap between the words a person uses and the meaning they intend is called the semantic gap ("semantic" just means "to do with meaning"). It includes synonyms, but also rephrasing ("what time do I need to be back"), abbreviations ("CGPA"), spelling mistakes and queries that mix English with Hindi, Telugu or Tamil. The more naturally people type, the bigger the gap, and people type very naturally into AI assistants.
Each pair means the same thing to a person, but shares almost no words, so a keyword search sees no connection.
One word, several meanings
The opposite problem also happens. Polysemy means one word with several meanings. "Leave" can mean medical leave or a holiday. "Break" can be a holiday or something that is broken. On the Kaveri pages, the query "internet not working" returns the Bonafide Certificate page, because stemming turns "working" into "work" and that page promises the certificate within three "working days". The words match; the meaning does not.
So keyword search fails in two opposite ways:
- Synonyms cause misses. The right page is never returned because it uses different words.
- Polysemy causes false hits. A wrong page is returned because it uses the same word with a different meaning.
Both come from the same cause. A word is only a pointer to a meaning, and keyword search compares the pointers, not the meanings.
Keyword search can only see whether words match. Two of the four boxes are mistakes it cannot detect.
Sort these real examples into the two kinds of failure.
What the person actually wants
Even when the words are right, two people can mean different things. A first-year in the hostel who types "fees" probably means hostel fees; a parent might mean tuition. "Last date" in July is probably about the fee instalment, and in August about scholarship applications. "Can I go home for Diwali?" is really two questions: is there a break, and does the hostel stay open?
This is about intent (what the person is trying to do) and context (who is asking, when, and what they asked a moment ago). A keyword ranking method sees none of this. It gets a few words and scores pages the same way for everyone, every time.
Fixing it by hand
Before computers could handle meaning, engineers patched the gap by hand. They wrote synonym lists ("vacation" also means "break"), added query expansion (automatically adding related words to a query), and built spelling correction and abbreviation lists. These patches work, and many systems still use them.
The lab below shows why they are also a lot of work to maintain. Pick a query, see whether keyword search finds the right page, then add words a person might think of and watch what happens, including to pages you did not mean to bring in.
Every query needed someone to predict the mismatch, write the synonym and check it did not break other searches. With thousands of documents and endless new questions, that list never ends.
To see just how little the words overlap, the program below compares each query with the page that answers it.
Expected output:
vacation rules shared words: none
hostel curfew shared words: ['hostel']
exam paper rechecked shared words: none
words() lowercases the text, drops stop words, strips a final "s" and returns a set. The & between two sets keeps only the words found in both, and that overlap is all a keyword search has to work with.
What a better search needs to do
These examples show what a better search must do. It should treat "sick days" and "medical leave" as close, even though they share no words, and treat "working days" and "not working" as far apart, even though they share one. In other words, it must compare the meaning of text, not its spelling. This is the idea behind semantic search. Google gave a famous example in 2019, when it started using a language model called BERT to understand searches: for "2019 brazil traveler to usa need a visa", the small word "to" changes everything, and keyword matching had been returning pages about Americans travelling to Brazil.
Keyword search does not become useless, though. It is still better when the exact text is what matters: roll numbers, course codes like "CS3101", error messages, form numbers and rare names. A student searching for "Form 16A" wants that exact form, not something similar. Good modern systems use both: keyword matching for exact terms, and meaning-based matching for everything else.