Production Readiness
Judge whether a RAG system is ready to launch using SLOs, fallbacks, rollbacks, load tests and runbooks.
Is it ready for real users?
A demo answers the questions you chose. A production system answers whatever thousands of people ask, at any hour, while documents change and dependencies fail. Production readiness is the set of qualities that make that possible: clear quality and speed targets, tested behaviour under load, safe ways to fail, a way to undo bad changes, and people who know what to do when something breaks. Checking readiness before launch, deliberately, avoids learning these lessons from angry users.
Service level objectives
A service level objective (SLO) is a measurable target for how well the system should work, agreed in advance. Typical SLOs for a RAG assistant:
| Area | Example SLO |
|---|---|
| Availability | 99.5% of requests get a response (not an error) each month |
| Speed | p95 answer time under 4 seconds |
| Quality | Faithfulness ≥ 0.90 on the regression set; must-pass questions 100% |
| Freshness | Document changes searchable within 24 hours (exam notices within 1 hour) |
| Safety | Zero answers built from content the user may not access |
SLOs turn vague expectations ("it should be fast and accurate") into numbers that monitoring can track and alerts can enforce, and they make trade-offs explicit: a 4-second target allows a reranker; a 1-second target may not.
Failing safely
Every dependency will fail at some point: the model API times out, the vector database restarts, a document sync breaks. A production system plans for this with fallbacks:
- Model fallback: if the main model is unavailable or slow, switch to a backup model or provider.
- Retrieval fallback: if vector search fails, fall back to keyword search; if all search fails, say so honestly.
- Graceful messages: "I can't check the documents right now; please try again shortly or contact the academic office" is far better than an error page, and far better than an answer made up without sources.
- Timeouts and retries: every external call has a time limit, and brief failures are retried a small number of times.
Each dependency has a planned fallback, and the last resort is an honest message, never an unsourced guess.
Releasing changes safely
Changes to prompts, models, chunking or indexes should be released like code changes:
- Regression tests and quality gates run before every release.
- Gradual rollout: send a small share of traffic (say 5 percent) to the new version first, compare metrics, then increase. Running old and new side by side on the same questions is sometimes called shadow testing.
- Rollback: keep the previous version ready, including the previous index, so a bad change can be undone in minutes.
- Versioning: record which prompt, model, index and document versions produced every answer, so problems can be traced.
Load testing
Load testing sends realistic traffic at the system, at and above the expected peak, to see what breaks. For a college, the peak may be the morning results are announced, or the week before fee deadlines. Load tests reveal slow queries, rate limits at the model provider, database connection limits and memory problems before real users find them.
Runbooks
A runbook is a short, practical document that says what to do when a specific thing goes wrong: "model API returning errors", "answers quoting outdated fees", "a confidential document appeared in an answer". Each entry lists how to recognise the problem, how to stop the damage quickly, how to fix it, and whom to inform. Runbooks turn a stressful incident into a checklist.
A readiness checklist
The program below scores a system against a short readiness checklist and lists what is missing before launch.
Expected output:
ready: 7/10
MISSING: Model and retrieval fallbacks in place
MISSING: Load test at 2x expected peak
MISSING: Runbooks for top 5 incidents
Python counts True as 1, so sum(checklist.values()) is the number of items that are done. The loop then prints every item whose value is False.
Seven of ten is not a launch: the missing items are exactly the ones that matter on a bad day, when traffic spikes or a provider goes down.
Bringing it together
A production-ready RAG system is not one clever technique. It is a well-measured pipeline with clear targets, tested behaviour, safe failure modes, reversible changes, and people prepared to run it. Try judging these launch situations.