recallA retrieval
field guide

Multilingual and Indic Retrieval

Build this skill

Recognise native-script, transliterated and code-mixed queries, and use multilingual embeddings for Indian-language search.

Students do not all search in English

At a college in Hyderabad, one student types "medical leave rules", another types "medical leave ki rules kya hai", and a third types "మెడికల్ లీవ్ ఎన్ని రోజులు" (Telugu for "how many days of medical leave"). All three want the same English policy page. A search system that only understands English serves the first student well and fails the other two.

India is a natural test of this problem. People regularly switch between English and an Indian language, write Indian languages in English letters, and mix both in the same sentence. Building retrieval that works for them needs a few ideas that English-only systems can ignore.

Four kinds of multilingual queries

Kind Example What makes it hard
Native script छुट्टी के नियम Different alphabet from the English documents
Transliterated chutti ke niyam Hindi words written in English letters, with no fixed spelling
Code-mixed hostel ka fee kitna hai Two languages in one sentence
Cross-lingual Telugu question, English document Query and answer are in different languages

Transliteration means writing words from one language in another language's alphabet. "Chutti", "chhutti" and "chuti" are all common spellings of the Hindi word for leave, so even exact keyword matching on transliterated text is unreliable. Code-mixing means switching languages inside one sentence, which is how many people actually talk and type. Cross-lingual retrieval means the query and the document are in different languages, which is the normal case when a college's rules are written in English.

What students typeछुट्टी के नियम (Hindi)chutti ke niyam (Hinglish)hostel ka fee kitna haiమెడికల్ లీవ్ ఎన్ని రోజులు (Telugu)What the documents sayMedical Leave policyMedical Leave policyHostel Fees: ₹96,000 a yearMedical Leave: up to 10 days

The documents are in English, but the questions arrive in many forms. Keyword search shares almost no words between the two columns.

Why keyword search breaks here

Keyword search needs shared words. A Telugu query and an English document share none. Transliterated queries share none either, unless the spelling happens to match. Stemming and stop-word lists are built for one language at a time, so a code-mixed query like "hostel ka fee kitna hai" keeps "ka", "kitna" and "hai" as if they were meaningful terms.

The program below asks the same thing four ways and counts the words each version shares with the English page.

python · runs in your browser

Expected output:

text
English          shared words: ['leave', 'medical']
Hindi            shared words: none
Telugu           shared words: none
Romanised Hindi  shared words: ['medical']

Each query and the page are split into sets of words, and & keeps the words found in both. Only the English query has enough overlap to rank the page. The romanised query matches on "medical" alone, by luck, and the two queries in Indian scripts match nothing at all.

The traditional workaround was to translate the query into English with a translation system, then run an ordinary English search. This still works and can be a good fallback, but it adds a step, adds delay, and translation mistakes become search mistakes.

How multilingual embeddings help

A multilingual embedding model is trained on text in many languages, including many pairs of sentences that mean the same thing in different languages. Training pulls each pair together, so the model learns a single shared space where meaning matters more than language. "Leave rules", "छुट्टी के नियम" and the Telugu equivalent all land near each other, and near the English policy page.

This makes cross-lingual search almost free: embed English documents once, and queries in any supported language can find them. Models such as BGE-M3, multilingual E5 and Jina's multilingual models cover Hindi, Bengali, Tamil, Telugu, Marathi and other Indian languages to varying degrees. Indian research groups such as AI4Bharat have also built datasets and models focused on Indian languages.

Compare the three approaches on the same query below.

Testing multilingual search properly

Multilingual quality varies a lot between languages, even inside one model. A model can be excellent in Hindi and weak in Odia, because it saw far less Odia text during training. Benchmarks help a little: MTEB includes some Indian languages, and there are multilingual retrieval benchmarks such as MIRACL. But your own test matters more.

Build your test set from what real users type, in the mix of languages and scripts they actually use. Include transliterated and code-mixed queries, because these are often the weakest. Report results per language, not just an overall average, so that a strong English score cannot hide a failing Tamil one.

Try sorting these queries by the kind of challenge they present.

Practical tips

A few habits make multilingual retrieval much more reliable. Keep a keyword search alongside the embeddings for things that do not translate: roll numbers, course codes, names and form numbers. Store the document's language as metadata, so you can filter or report by language. When documents exist in several languages, such as an English and a Hindi version of the same circular, store both and link them, rather than relying on cross-lingual matching alone. And if one language performs badly in your tests, try translating just those queries into English as a fallback, rather than abandoning a model that works well for everything else.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close