recallA retrieval
field guide

Caching for RAG

Build this skill

Add embedding, retrieval and response caches safely, and decide when a cached answer may be reused.

Don't do the same work twice

In a college helpdesk, the same questions arrive again and again: "when is the hostel fee due?", "what is the library fine?", "how many days of medical leave?". Each time, a naïve system embeds the question, searches the index, reranks, builds a prompt and pays a language model to write an answer it has already written many times. Caching stores the results of expensive work so that repeated work can be skipped. For RAG, caching can cut costs and response times dramatically.

A cache is a fast store of recent results, looked up by a key. If the key is found (a cache hit), the stored result is used. If not (a cache miss), the work is done and the result is stored for next time. The share of lookups that hit is the hit rate.

Three levels of cache

Response cachequestion → final answer (skips everything)Retrieval cachequery → retrieved chunks (skips search and rerank)Embedding cachetext → embedding (skips the embedding call)

Higher levels save more work per hit, but are trickier to keep correct.

An embedding cache stores the embedding for each piece of text. It is completely safe, because the same text with the same model always gives the same embedding. It saves time and money on repeated queries and, especially, when re-indexing documents that did not change.

A retrieval cache stores the retrieved chunks for a query. It skips search and reranking, which are usually cheap, so it helps most when retrieval is slow or uses expensive steps such as query rewriting.

A response cache stores the final answer for a question. It saves the most, because generation is usually the slowest and most expensive step, but it also carries the most risk: a cached answer can become wrong when documents change, and it must never be shown to someone who is not allowed to see the sources it used.

Semantic caching

An exact cache only hits when the question text matches exactly. But "When is the hostel fee due?" and "hostel fee due date?" mean the same thing. A semantic cache looks up questions by meaning: it embeds the new question and checks whether a previous question's embedding is very close (above a similarity threshold). If so, it reuses that answer.

The threshold is crucial. Too low, and "hostel fee for first-years" might return the cached answer for "hostel fee for seniors", which is wrong. Too high, and the cache rarely hits. Set it by testing pairs of questions that should and should not share an answer.

The program below implements a tiny semantic cache using word overlap as a stand-in for embedding similarity.

python · runs in your browser

Expected output:

text
hostel fee due date?           HIT  (0.75) In two instalments, July and January [Hostel Fees 2026].
When is the hostel fee due?    HIT  (1.00) In two instalments, July and January [Hostel Fees 2026].
What is the library fine?      MISS (0.00) -> run the full RAG pipeline
When is the exam fee due?      MISS (0.50) -> run the full RAG pipeline

similarity divides the words two questions share by all the distinct words in either. ask finds the cached question most similar to the new one and reuses its answer only if that similarity reaches THRESHOLD.

The last line is the important one: "exam fee due" shares several words with "hostel fee due" but correctly misses. With a lower threshold, it would have returned the hostel answer to a question about exam fees. Find the safe range yourself.

Keeping cached answers correct

Invalidation means removing cached results that are no longer correct. For RAG, the main triggers are:

  • Document changes: when a source document is updated or deleted, remove cached answers that used it. Storing the source IDs with each cached answer makes this possible.
  • Time limits: give every cached answer an expiry (a time to live, or TTL), short for fast-changing topics (exam notices) and longer for stable ones (library rules).
  • System changes: a new prompt, model or index version should clear the response cache.

Permissions need equal care. Cache keys should include the user's access level (or the cache should be per role), so an answer built from staff-only documents is never served to a student. Tools such as Redis, an in-memory data store, are commonly used for caches because they are very fast and support expiry times, and Redis also offers vector search features that can back a semantic cache.

What caching saves

Caching is one of the highest-value optimisations for systems with many repeated questions, which describes most helpdesks. Start with an embedding cache (safe), add an exact response cache with short expiry and document-based invalidation, and only then consider a semantic cache, carefully tested.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close