recallA retrieval
field guide

Production Readiness

Build this skill

Judge whether a RAG system is ready to launch using SLOs, fallbacks, rollbacks, load tests and runbooks.

Is it ready for real users?

A demo answers the questions you chose. A production system answers whatever thousands of people ask, at any hour, while documents change and dependencies fail. Production readiness is the set of qualities that make that possible: clear quality and speed targets, tested behaviour under load, safe ways to fail, a way to undo bad changes, and people who know what to do when something breaks. Checking readiness before launch, deliberately, avoids learning these lessons from angry users.

Service level objectives

A service level objective (SLO) is a measurable target for how well the system should work, agreed in advance. Typical SLOs for a RAG assistant:

Area Example SLO
Availability 99.5% of requests get a response (not an error) each month
Speed p95 answer time under 4 seconds
Quality Faithfulness ≥ 0.90 on the regression set; must-pass questions 100%
Freshness Document changes searchable within 24 hours (exam notices within 1 hour)
Safety Zero answers built from content the user may not access

SLOs turn vague expectations ("it should be fast and accurate") into numbers that monitoring can track and alerts can enforce, and they make trade-offs explicit: a 4-second target allows a reranker; a 1-second target may not.

Failing safely

Every dependency will fail at some point: the model API times out, the vector database restarts, a document sync breaks. A production system plans for this with fallbacks:

  • Model fallback: if the main model is unavailable or slow, switch to a backup model or provider.
  • Retrieval fallback: if vector search fails, fall back to keyword search; if all search fails, say so honestly.
  • Graceful messages: "I can't check the documents right now; please try again shortly or contact the academic office" is far better than an error page, and far better than an answer made up without sources.
  • Timeouts and retries: every external call has a time limit, and brief failures are retried a small number of times.
Question arrivesRetrieval OK?noKeyword fallbackif vectors failMain model OK?noBackup modelwithin timeoutEverything down?yesHonest messagewho to contact

Each dependency has a planned fallback, and the last resort is an honest message, never an unsourced guess.

Releasing changes safely

Changes to prompts, models, chunking or indexes should be released like code changes:

  • Regression tests and quality gates run before every release.
  • Gradual rollout: send a small share of traffic (say 5 percent) to the new version first, compare metrics, then increase. Running old and new side by side on the same questions is sometimes called shadow testing.
  • Rollback: keep the previous version ready, including the previous index, so a bad change can be undone in minutes.
  • Versioning: record which prompt, model, index and document versions produced every answer, so problems can be traced.

Load testing

Load testing sends realistic traffic at the system, at and above the expected peak, to see what breaks. For a college, the peak may be the morning results are announced, or the week before fee deadlines. Load tests reveal slow queries, rate limits at the model provider, database connection limits and memory problems before real users find them.

Runbooks

A runbook is a short, practical document that says what to do when a specific thing goes wrong: "model API returning errors", "answers quoting outdated fees", "a confidential document appeared in an answer". Each entry lists how to recognise the problem, how to stop the damage quickly, how to fix it, and whom to inform. Runbooks turn a stressful incident into a checklist.

A readiness checklist

The program below scores a system against a short readiness checklist and lists what is missing before launch.

python · runs in your browser

Expected output:

text
ready: 7/10
  MISSING: Model and retrieval fallbacks in place
  MISSING: Load test at 2x expected peak
  MISSING: Runbooks for top 5 incidents

Python counts True as 1, so sum(checklist.values()) is the number of items that are done. The loop then prints every item whose value is False.

Seven of ten is not a launch: the missing items are exactly the ones that matter on a bad day, when traffic spikes or a provider goes down.

Bringing it together

A production-ready RAG system is not one clever technique. It is a well-measured pipeline with clear targets, tested behaviour, safe failure modes, reversible changes, and people prepared to run it. Try judging these launch situations.

Follow your curiosity.

↑ ↓ to explore · Enter to open · Escape to close