recallA retrieval
field guide

Conversational RAG

Build this skill

Handle follow-up questions by rewriting them into standalone queries and managing chat history.

Follow-up questions break simple retrieval

People do not ask questions in isolation. A real conversation with a campus assistant looks like this:

Student: How many days of medical leave can I take? Assistant: Up to 10 days per semester; more than that needs the Dean of Academics [1]. Student: And for hostel students? Student: What about the certificate for that?

A naïve RAG system searches with each message on its own. "And for hostel students?" has no mention of medical leave, so retrieval returns hostel timings and hostel fees. "What about the certificate for that?" is even worse: "that" refers to something two messages back. The model ends up answering a different question from the one the student asked.

Conversational RAG handles this by using the conversation's history when searching and when answering.

Chat history

The chat history is the list of previous messages in the conversation. It is useful in two places:

  • Before retrieval, to work out what the latest message actually means.
  • During generation, so the model can keep its answers consistent with what it already said.

Simply pasting the whole history into the search query does not work well. Long, mixed-topic text produces a vague embedding and pulls in passages from earlier, unrelated parts of the conversation.

Rewriting the question so it stands alone

The standard technique is standalone question rewriting (also called query condensation). Before searching, a language model is given the chat history and the latest message, and asked to rewrite the message as a complete question that makes sense on its own. Retrieval then uses the rewritten question.

Latest message Rewritten standalone question
And for hostel students? How many days of medical leave can hostel students take?
What about the certificate for that? What medical certificate is needed for medical leave, and by when?
Who do I submit it to? Who should a student submit the medical leave certificate to?

The rewritten question is used for searching; the answer is still written for the student's actual message, with the conversation in view.

Latestmessage“what aboutthat?”Rewriteusing chathistoryStandalonequestioncomplete andsearchableRetrieveAnswerwith history +sources

The history is used to turn a follow-up into a complete question before searching.

Step through a short conversation below and compare what gets searched with and without rewriting.

Memory over longer conversations

Conversations can run for many turns, and the history keeps growing. Sending the full history with every request eventually costs too many tokens, and old turns can confuse the model. Common approaches are to keep only the last few turns, or to keep a short running summary of older turns plus the most recent messages in full.

A few more details matter in practice:

  • Topic changes. When the student switches from leave to library fines, the rewriter should not drag old topics into the new question.
  • Privacy. Chat history may contain personal details. Store it only as long as needed, keep it separate per user, and never retrieve one user's history for another.
  • Stale answers. If the knowledge base changes during a long conversation, earlier answers in the history may now be wrong; it is safer to retrieve fresh sources each turn than to rely on what was said before.

The program below implements a very simple rewriter that fills in what pronouns like "that" refer to, using the topic of the previous question. A real system uses a language model for this, but the input and output are the same.

python · runs in your browser

Expected output:

text
And for hostel students?                 -> How many days of medical leave can hostel students take?
What about the certificate for that?     -> What is required regarding the certificate for medical leave?

last_topic looks at the previous question and returns the subject it was about. rewrite then swaps the vague phrase in the new message ("for that", or a bare "And for ...") for that subject, producing a question that makes sense without the conversation. The rewritten question is what gets searched, not the original message.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close