recallA retrieval
field guide

Choosing a Retrieval Architecture

Build this skill

Choose and grow a retrieval architecture from the mix of real question types, measuring each addition.

Pick the simplest design that answers the questions

There are now many ways to build a system that answers questions from documents: naïve RAG, hybrid search with reranking, contextual retrieval, knowledge graphs and GraphRAG, text-to-SQL, long context and caching, multimodal retrieval, and agents that combine several of these. It is tempting to reach for the most sophisticated option. In practice, the best architecture is the simplest one that answers your users' real questions well enough, at a cost and speed you can sustain. Every extra component adds cost, delay, things to maintain and new ways to fail.

Choosing well starts with the questions, not the technology.

Start from the questions people ask

Collect a sample of real questions (from search logs, helpdesk emails, or by asking users) and sort them into types. The mix of types tells you what the architecture must handle.

Question type Example Architecture that fits
Single fact in one passage "What is the library fine?" Naïve or hybrid RAG
Exact codes, names or numbers "CS3101 syllabus" Hybrid search (keywords + vectors)
Fact spread over a few passages "Who approves extra leave, and what paperwork?" Hybrid + reranking, multi-query
Counts, sums, comparisons over records "How many CSE students were placed?" Text-to-SQL over a database
Chains of relationships "Is the Block C warden on the committee?" Knowledge graph or agent
Themes across the whole collection "Main grievance themes this year?" GraphRAG or long-context summarisation
Answers in charts and page layouts "What does the placement chart show for 2025?" Multimodal retrieval
Small, stable reference set "Anything in the 50-page staff handbook" Long context with caching
Needs no documents "Hi!" / "Thanks" No retrieval

Most real systems face a mix. A common, robust result is a hybrid RAG pipeline with reranking as the core, plus one or two specialised routes, such as text-to-SQL for statistics, chosen by a router.

A decision path

What do mostquestions need?Facts in passagesHybrid RAG +rerankthe default coreNumbers overrecordsText-to-SQLexact countsLinks or themesGraph / GraphRAGrelations, summariesSmall, stable setLong contextwith caching

Start from the dominant question types. Add routes for the others only when tests show they are needed.

Weighing the trade-offs

Every architecture can be judged on the same five criteria. The table below gives a rough comparison; your own measurements should replace it.

Naïve RAG Hybrid + rerank GraphRAG Text-to-SQL Long context Agentic
Build effort Low Medium High Medium Very low High
Cost per question Low Low–medium Medium–high Low High (unless cached) High
Speed Fast Fast Slow for global Fast Slow for big inputs Slow
Handles multi-hop / global No Partly Yes For data only Partly Yes
Easy to debug Yes Yes Harder Yes (query is visible) Hard Harder

Grow the architecture step by step

A reliable way to get to the right design:

  1. Build the baseline: hybrid search, a sensible chunker, a good prompt with citations.
  2. Build a test set from real questions, labelled by type, with known answers and sources.
  3. Measure, and break results down by question type.
  4. Add one component aimed at the worst-performing type: a reranker, multi-query, a SQL route, a graph.
  5. Measure again. Keep the component only if it clearly helps that type without hurting others or blowing the budget.

The program below shows step 3 and 4 in miniature: accuracy by question type for a baseline, and the effect of adding a SQL route for statistics questions.

python · runs in your browser

Expected output:

text
single fact  baseline 92%   + SQL route 92%
spread out   baseline 70%   + SQL route 70%
statistics   baseline 24%   + SQL route 88%
multi-hop    baseline 33%   + SQL route 33%

overall: baseline 62%  ->  with SQL route 78%

overall is a weighted average: each type's accuracy is multiplied by the number of questions of that type, and the total is divided by the number of questions. {**baseline, "statistics": 0.88} copies the baseline and changes one entry, so the two columns differ only where the SQL route applies.

The weakest type, statistics, was also a large share of questions, so a SQL route raised overall accuracy far more than any tweak to the text pipeline would have. Multi-hop is now the obvious next target.

Practise the decision

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close