Reranking Retrieved Results
Rerank retrieved candidates with a cross-encoder and tune how many to retrieve and keep.
A second, more careful look
Vector search is fast because it compares pre-computed embeddings: the query and each passage were turned into numbers separately, and search just measures distances. That speed comes at a cost. The query and the passage never "see" each other, so fine details are lost: whether the passage actually answers the question, whether it is about students or staff, whether "10 days" refers to leave or to the re-evaluation deadline.
Reranking adds a second step. Retrieval quickly fetches a generous set of candidates, say the top 30. Then a slower, more accurate model, the reranker, reads the question together with each candidate and gives each pair a new relevance score. The candidates are re-sorted by that score, and only the best few go into the prompt. This "retrieve many, rerank, keep few" pattern is one of the most reliable ways to improve a RAG system.
Retrieval is tuned for recall; reranking is tuned for precision on a short list.
Bi-encoders and cross-encoders
The difference comes down to how the two kinds of model read text.
A bi-encoder (used for retrieval) encodes the question and each passage separately into embeddings and compares them. Passages can be encoded ahead of time, which makes search over millions of passages possible.
A cross-encoder (used for reranking) reads the question and one passage together, as a single input, and outputs one relevance score. Because every word of the question can be compared directly with every word of the passage, it judges relevance much more precisely. But nothing can be computed ahead of time: scoring 30 candidates means running the model 30 times for each question. That is fine for 30 passages and impossible for 3 million, which is why cross-encoders are used only on the short list that retrieval provides.
| Bi-encoder (retrieval) | Cross-encoder (reranking) | |
|---|---|---|
| Reads question and passage | Separately | Together |
| Can pre-compute passages | Yes | No |
| Speed over millions | Fast | Impossible |
| Precision | Good | Better |
| Typical use | Find the top 20–100 candidates | Re-sort them, keep the top 3–5 |
Reranking options
Several kinds of reranker are common:
- Open cross-encoder models such as the BGE reranker family from BAAI, which can run on your own servers and support many languages.
- Hosted rerank APIs from providers such as Cohere, Jina and Voyage AI: send the question and candidates, receive scores.
- Language models as rerankers: ask a general language model to score or order the candidates. This can be very accurate but is slower and more expensive, so it is usually reserved for small candidate lists.
See the effect below. Retrieval has returned six candidates; the prompt has room for three. Switch reranking on and off and count how many useful passages make it into the prompt.
Without reranking, only one useful passage made the top three; the staff rule and the attendance rule, which merely mention medical leave, crowded the second useful passage out. The reranker, reading each passage with the question, recognised that "a student" and "who approves" were the point.
Using a reranker in code
With the sentence-transformers library, an open cross-encoder takes a few lines. This runs on your own computer after installing the package and downloading the model:
# pip install sentence-transformers
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
question = "How many days of medical leave can a student take?"
candidates = [doc.page_content for doc in retrieved_docs] # e.g. the top 30
scores = reranker.predict([(question, c) for c in candidates])
best = [c for _, c in sorted(zip(scores, candidates), reverse=True)[:4]]
The structure is always the same, whichever reranker you use. The program below shows it with a stand-in scoring function, so you can see the re-sorting step in isolation.
Expected output:
retrieval order: ['Medical leave is added back when at', 'Faculty may take 20 days of medical']
reranked order: ['Students may take up to 10 days of ', 'Faculty may take 20 days of medical']
sorted(candidates, key=lambda c: -toy_cross_encoder(question, c[0])) is the whole reranking step: score every candidate again with the slower, more careful function, and sort by the new score. The retrieval scores are ignored. The student rule, which retrieval had placed third, moves to the top.
Tuning the pattern
Two numbers matter: how many candidates to retrieve, and how many to keep. Retrieving more gives the reranker more chances to find the right passage but costs more reranking time; keeping more uses more prompt space. A common starting point is to retrieve 20 to 50 and keep 3 to 8, then adjust using Recall@K on your test questions and the latency you can afford. A reranker typically adds tens to a few hundred milliseconds, depending on the model, the hardware and the number of candidates.
Reranking does not rescue a passage that retrieval never found. If the right passage is not in the top 30, no reranker can bring it back; that is a retrieval or chunking problem. What reranking does is make sure that, when the right passage is anywhere in the candidate list, it is the one the model reads.