Query Routing
Route questions to the right collection, database or response, with fallbacks for uncertain cases.
Not every question should go to the same place
A campus assistant receives very different kinds of questions in one afternoon:
- "What is the medical leave policy?" — answered by the regulations.
- "What is my attendance in CS3101?" — answered by the student database, not by any document.
- "When is the next placement drive?" — answered by the placement cell's notices.
- "Hi, thanks!" — needs no search at all.
- "Write my assignment on operating systems" — should be politely declined.
Sending all of them through the same vector search over one big index gives poor results: personal data questions find nothing, small talk triggers pointless searches, and policy questions compete with placement notices. Query routing decides, for each question, where it should go: which collection, which tool, or no retrieval at all.
What a router chooses between
A router sits at the front of the system and picks a route for each question. Typical routes are:
| Route | Example question | What happens |
|---|---|---|
| A document collection | "medical leave rules" | Vector or hybrid search in the regulations collection |
| Another collection | "next placement drive" | Search the placement notices collection |
| A database or API | "my attendance in CS3101" | Run a query against the student information system |
| No retrieval | "thanks!" / "hi" | Reply directly |
| Out of scope | "write my assignment" | Decline politely, explain what the assistant can help with |
| Ask a clarifying question | "fees?" | Ask: "Hostel, tuition or exam fees?" |
One entry point, several destinations. The router decides which one each question needs.
How routers decide
There are three common ways to build a router, from simplest to most flexible.
Rules
Keyword patterns or the part of the app the user is in: questions asked from the Placements page go to placement notices; questions containing "my attendance" go to the database. Rules are fast and predictable, but brittle with unexpected phrasing.
Embedding similarity
Write a few example questions for each route, embed them, and send each new question to the route whose examples are closest. This handles paraphrases well and needs no extra model calls beyond the embedding.
A language model classifier
Describe each route to a language model and ask it to choose, often with structured output such as {"route": "student_database"}. This is the most flexible and handles complex questions, at the cost of an extra model call per question.
Many systems combine them: cheap rules for the obvious cases, and a model for the rest.
The program below builds a small embedding-style router using word overlap with example questions, and routes a few new questions.
Expected output:
What is my attendance in CS3101? -> student_database
Which company is visiting campus this week? -> placements
Rules for medical leave? -> regulations
thanks! -> small_talk
fees? -> ask_clarifying_question
For each route, s is the best match among its example questions, measured as the fraction of an example's words that also appear in the new question. The route with the highest s wins. The last line is the safety net: if even the best score is under 0.5, the router does not guess and asks the user to clarify, which is what happens to the one-word question "fees?".
When the router is unsure
Routers make mistakes, and some questions genuinely fit several routes. Three safeguards help. Confidence thresholds: if no route scores clearly highest, ask the user a short clarifying question rather than guessing. Fallbacks: if the chosen collection returns nothing relevant, try a broader search before saying "I don't know". Multiple routes: for questions that span sources ("compare my attendance with the requirement"), send them to both the database and the regulations, and combine the results.
Most importantly, routing must never be a way around security. A question routed to the student database must only ever see the logged-in student's own records, whatever the question says.
Try routing these questions.