Choosing a Retrieval Architecture
Choose and grow a retrieval architecture from the mix of real question types, measuring each addition.
Pick the simplest design that answers the questions
There are now many ways to build a system that answers questions from documents: naïve RAG, hybrid search with reranking, contextual retrieval, knowledge graphs and GraphRAG, text-to-SQL, long context and caching, multimodal retrieval, and agents that combine several of these. It is tempting to reach for the most sophisticated option. In practice, the best architecture is the simplest one that answers your users' real questions well enough, at a cost and speed you can sustain. Every extra component adds cost, delay, things to maintain and new ways to fail.
Choosing well starts with the questions, not the technology.
Start from the questions people ask
Collect a sample of real questions (from search logs, helpdesk emails, or by asking users) and sort them into types. The mix of types tells you what the architecture must handle.
| Question type | Example | Architecture that fits |
|---|---|---|
| Single fact in one passage | "What is the library fine?" | Naïve or hybrid RAG |
| Exact codes, names or numbers | "CS3101 syllabus" | Hybrid search (keywords + vectors) |
| Fact spread over a few passages | "Who approves extra leave, and what paperwork?" | Hybrid + reranking, multi-query |
| Counts, sums, comparisons over records | "How many CSE students were placed?" | Text-to-SQL over a database |
| Chains of relationships | "Is the Block C warden on the committee?" | Knowledge graph or agent |
| Themes across the whole collection | "Main grievance themes this year?" | GraphRAG or long-context summarisation |
| Answers in charts and page layouts | "What does the placement chart show for 2025?" | Multimodal retrieval |
| Small, stable reference set | "Anything in the 50-page staff handbook" | Long context with caching |
| Needs no documents | "Hi!" / "Thanks" | No retrieval |
Most real systems face a mix. A common, robust result is a hybrid RAG pipeline with reranking as the core, plus one or two specialised routes, such as text-to-SQL for statistics, chosen by a router.
A decision path
Start from the dominant question types. Add routes for the others only when tests show they are needed.
Weighing the trade-offs
Every architecture can be judged on the same five criteria. The table below gives a rough comparison; your own measurements should replace it.
| Naïve RAG | Hybrid + rerank | GraphRAG | Text-to-SQL | Long context | Agentic | |
|---|---|---|---|---|---|---|
| Build effort | Low | Medium | High | Medium | Very low | High |
| Cost per question | Low | Low–medium | Medium–high | Low | High (unless cached) | High |
| Speed | Fast | Fast | Slow for global | Fast | Slow for big inputs | Slow |
| Handles multi-hop / global | No | Partly | Yes | For data only | Partly | Yes |
| Easy to debug | Yes | Yes | Harder | Yes (query is visible) | Hard | Harder |
Grow the architecture step by step
A reliable way to get to the right design:
- Build the baseline: hybrid search, a sensible chunker, a good prompt with citations.
- Build a test set from real questions, labelled by type, with known answers and sources.
- Measure, and break results down by question type.
- Add one component aimed at the worst-performing type: a reranker, multi-query, a SQL route, a graph.
- Measure again. Keep the component only if it clearly helps that type without hurting others or blowing the budget.
The program below shows step 3 and 4 in miniature: accuracy by question type for a baseline, and the effect of adding a SQL route for statistics questions.
Expected output:
single fact baseline 92% + SQL route 92%
spread out baseline 70% + SQL route 70%
statistics baseline 24% + SQL route 88%
multi-hop baseline 33% + SQL route 33%
overall: baseline 62% -> with SQL route 78%
overall is a weighted average: each type's accuracy is multiplied by the number of questions of that type, and the total is divided by the number of questions. {**baseline, "statistics": 0.88} copies the baseline and changes one entry, so the two columns differ only where the SQL route applies.
The weakest type, statistics, was also a large share of questions, so a SQL route raised overall accuracy far more than any tweak to the text pipeline would have. Multi-hop is now the obvious next target.